FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

How to: format/float precision?

 
Post new topic   Reply to topic     Forum Index -> General
View previous topic :: View next topic  
Author Message
Matthias



Joined: 18 Sep 2007
Posts: 6

PostPosted: Sat Apr 04, 2009 10:36 pm    Post subject: How to: format/float precision? Reply with quote

Beginner's question here, as I usually just make small applications to help me do something. I have a program that takes a number from a text box and adds it to another number, then spits it out as text. I noticed that sometimes they were getting cut off, and after poking around I traced it to doFormat's default of six significant digits. This is the relevant line, how would I go about raising the precision?

Code:

newSeconds = std.string.toString(atof(seconds) + displacement);
Back to top
View user's profile Send private message
csauls



Joined: 27 Mar 2004
Posts: 278

PostPosted: Sun Apr 05, 2009 6:27 pm    Post subject: Reply with quote

Code:
newSeconds = std.string.format("%13.9f", std.conv.toReal(seconds) + displacement);


The magic is in "%13.9f". The first number is the number of characters to emit, the second is how many should go after the decimal point. So, in this case, it emits three for the whole part, then the dot, then nine for the fraction (3 + 1 + 9 = 13).
_________________
Chris Nicholson-Sauls
Back to top
View user's profile Send private message AIM Address Yahoo Messenger
Matthias



Joined: 18 Sep 2007
Posts: 6

PostPosted: Mon Apr 06, 2009 9:36 pm    Post subject: Reply with quote

Awesome, thanks a lot.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> General All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group