HOME - - - - - - - TUTORIALS INDEX - - - - - - - - - - - - Other material for programmers

Delphi: Extracting data from records in text files

Objectives:

First I'll explain what I wanted a program for. You can decide whether there will be elements of interest to you! I'll also be taking you through my development process, which I think illustrates important skills.

I am a stock market investor. At the end of each day, I use a program called Personal Stock Monitor (link to their site) to fetch data for a number of stocks. That program maintains text files, one for each stock I follow, with a line for each day's data. They are in chronological order. The following is an example. It was a happy accident that I turned to this program just after my data supplier changed the format of the data slightly! The files do not have the column headings.... 
Date		Time	Prices					Volume		Prices
                                         Close     High        Low
09/26/2003	21:53	18.4210	19.0500	18.3600	29403008	18.7500	11.8400	24.9900
10/06/2003	22:35	19.3100	19.6100	19.1000	17971696	19.5300	19.3000	19.3400
10/09/2003	22:57	20.5530	20.8200	20.1690	38441856	20.5900	20.5900	20.6400
10/21/2003	22:24	21.89	21.97	21.58	19685136	21.96	21.81	21.87
10/29/2003	03:46	22.55	22.60	21.34	47612796	21.42	11.58	33.41
10/31/2003	00:44	23.24	23.48	22.58	34529904	22.98	12.09	34.39
10/31/2003	00:44	23.24	23.48	22.58	34529904	22.98	12.09	34.39
All of the text files are named with the standard stock market abbreviation for the stock. They are all in one folder. What I wanted was a program to scan all of these data files and produce a single file telling me the prices for all the different companies on a given day.... typically a recent day. I.e., a file like the following should be generated from the data in APC.txt, BORL.txt, CMX.txt, etc. (These are ficticious prices)

ABS 12.43 APC 45.25 BORL 8.94 CMX 56.24 EMC 28.41 ... etc

(This file will be used by other programs and used to supply the data to spreadsheets... but that is outside of the scope of this tutorial.)

Anyone using the program regularly would soon become annoyed if he/ she had repeatedly to enter the path to the folder with the datafiles. Additionally, I would think it reasonable to allow the user to specify where the output file should go, and what name to give it just once. Information like this is, in "proper" Windows programming supposed to go in the registry... a practice I dislike for various reasons. I still use the "old fashioned" approach: I keep such information in an "ini" (for "initialization") file... but that refinement is also outside this tutorial's scope.

Any time you set out to write a program, think long and hard about where you are going before you start. Know what the program will do, in detail. Start by thinking about the user's experience. What should the profram produce? What input will be needed? What options will exist? Even think in detail about the appearance of the forms which will appear on the screen. Think about what information needs to be tracked while the program runs. In a similar vein, think about what the program's output is going to be. Lastly, you have to think how you will accomplish the things that have to go on behind the scenes. You may have to revise your objectives at this stage!

From part of what I just said, it would seem that I should insert a graphic into this tutorial, to show you the basic form that the user will encounter when running the program. I agree... but... forgive me?.... The writing of these tutorials is quite a bit of work, and starting to create graphics is just one chore I'm shirking. I think the descriptions that follow should suffice. Perhaps this is also the place to apologise: This tutorial needs some editing to make it more readable. The information is, I hope, correct. I thought you'd rather have imperfect now vs. better much later?

Project's parts:

The project presents a number of interesting problems. They could be tackled in a variety of orders. Knowing which bit to start with is part of the art of programming. As I said above, be sure that you know how you are going to do all parts of the project before you start, but, once you have those plans laid, start "building your house" by assembling sub-units that can eventually be bolted together. This tutorial illustrates that approach, I hope.

Project's parts:
Providing for the use of the ini file to "fill in" default requirements.
Opening, writing to, closing the output file.
Getting initial information from the user (e.g. What is the date for which prices are required.)
Looking at one file, to see if a price date is available for that stock.
Besides being able to fetch the detail from one file of prices, the mechanism to scan a bunch of prices files needs to be created.

Let's start...

I'm going to start the project by creating a program which will go to a specific price file and look for prices for a particular date. I'm working in Delphi 2 for this tutorial.

Parts of what I suggest you type in the following are not "needed" in the final program. They are here so that you will experience the route I took to the program I needed. The tutorial was merely a bonus added to obtaining a real program I needed for a real job. If you follow the tutorial through, you see how a program can be developed without undue headache. The final result is far from being the whole point.

Speaking of headache: I'm sorry there are so many fields in each of the data records. Three or four would have done for the purposes of the tutorial, but the data records of my real world need were what you see here.

You'll find the following tutorials also address the questions of reading and writing to data files. They were written assuming less familiarity with the concepts that is assumed here.
(Level 2) File Handling... How to read data from files on disc, and write to such files. (A long tutorial)
(Level 3) How to access database files... It is remarkably easy to write a program in Delphi which allows you to view and edit files shared with Paradox, dBase, Access, etc. Learn how here!
Images and File Access... Display .bmp images on your form. Access all the files in a given folder on your disc, using that as the basis for a "Can you recognise..." exercise. Tutorial has rough edges, but full source listing of working program given.

Time to start typing...

Create a folder for the project.
Create a folder within it called DemoData... to hold some files with price information.
Use Wordpad to create a file holding the data given above ("09/26/2003 21:53 18.4...") The data are separated by tabs within each line, and a CR/LF pair at the end of each line.
Save that data several times, as: IBM.txt, PFE.txt, and MCD.txt... to simulate files for prices of IBM, Pfizer, and McDonalds. (Yes, that really is McDonalds' ticker. Pity. (Think about it... extra credit to those who email (text only!) with the reason why!)) Change the prices if you want to. Or the names, for that matter! These files are just for seeing if we can access some test data, before trying to deal with "the real thing".

Start a new project. I called mine DD42 (42nd Delphi Demo program)
Name the form DD42f1.
Put a button called buDoIt, captioned 'Do It', on the form.
Put a lable on the form called laTmp.

Create the following event handler for buDoItClick: 
procedure TDD42f1.buDoItClick(Sender: TObject);
begin
laTmp.caption:='At '+DateTimeToStr(time)+', button works';
end;
 Save the unit and project as DD42u1 and DD42 respectively.

Hardly a finished project... but a start! Our "base camp" is going to be a program which will report the price for a given stock on a given date when we click the "Do It" button.

Pressing on....



Just after the "private" in the TDD42f1 class definition, declare the following variables: 
    boDFInOpen,boDFOutOpen:boolean;
    dfIn:file of byte;
    dfOut:text;
    sTmp:string;
    bTmp:byte;
    liTmp:longint;
The first two will be flags which we use to keep track of whether an input data file is open and whether the output datafile is open. You might wonder, "Well, don't we know?" We know... but will your program know? Suppose a user decides to click the little "x" in the form's upper right hand corner while the program is struggling through all the datafiles? We won't be using the output file for a bit, but it was just easier to get those variables in place now.)

Create a function as follows.... 
function TDD42f1.ReadFromByteFile(liStrt:longint):string;
begin
ReadFromByteFile:='faked ReadFromByteFile';
end;
 Put.... 
function ReadFromByteFile(liStrt:longint):string;
 ... in the TDD42f1 class definition, just before the word "public"

Revise DoItClick to make it.... 
begin
assignfile(dfIn,'DemoData/IBM.txt');
reset(dfIn);
boDFInOpen:=true;
sTmp:=ReadFromByteFile(1);
closefile(dfIn);
boDFInOpen:=false;
laTmp.caption:=sTmp;
end;
 ... and get that much "working". (It won't actually read from the file yet, but get the typos out of that much first!)

Click on your form, go to the Object inspector (by pressing F11, or other way), set the OnClose event handler to.... 
if boDFInOpen=true then closefile(dfIn);
 This would be a good time, if you are inclined, to give yourself a "Quit" button which has.... 
application.terminate;
 ... for its OnClick event handler.

Time to read some data

Now.. let's actually read some data from the file. If you had small files, or if you only wanted to read things near the start of a file, you could do things much more simply. Sadly, I have big files, and will typically want to read things from near their end. To accomodate those needs, modify the ReadFromByteFile function to create the following: 
function TDD42f1.ReadFromByteFile(liStrt:longint):string;
begin
sTmp:='';
repeat
  seek(dfIn,liStrt);(*Move file's pointer to a place in file*)
  read(dfIn,bTmp);
  sTmp:=chr(bTmp)+sTmp;
  dec(liStrt);
  until ((bTmp=10)or(bTmp=13)or(liStrt=-1));
  if liStrt>-1 then sTmp:=copy(sTmp,2,length(sTmp)-1);
result:=sTmp;
end;
 ... and add... 
liTmp:=filesize(dfIn)-1;
sTmp:=ReadFromByteFile(liTmp);
 ... to buDoItClick, replacing the ... sTmp:=ReadFromByteFile(1); .. which was there previously.

Be sure your form is quite wide. If it isn't, you may think ReadFromByteFile is failing to read part of the line. Something else to look out for: If your data file has a blank line at the end, or even just an extra "return", it will look as if "Do It" did nothing.

An aside: Using liTmp to keep track of where we are in the file made sense at this point in the programming. Later, the value in liTmp was needed over a wider and wider scope, and it became quite badly exposed to inadvertant changes when you thought the old value was no longer important. If it were not for the hassle of changing all of the program code AND the text of this tutorial, I would have created and used a more meaningfully named variable to track our place in the file. Moral of the story: Use variable named "Tmp" carefully. ("Tmp" or "Temp" for temporary. If you are ever doing work with temperatures, use "Tture" for variables holding those values!)(Changing liTmp to liPlaceInFile within the Delphi... even if you don't want to change all references to liTmp... is quite easy because Delphi is well written.)

Have data, must parse

Parse: Break record into separate fields.

So far, so good... I hope. You should by now have a program that will read the last line in a data file.

Next we're going to add a function to parse the data line that has been read. With care, we can put almost everything we need to adapt the program for different data file format in just this and a few other procedures.

First put the following into your program just before the existing "type.. TDD42f1 = class(TForm).." 
type
  TParsed = record
      sErr,sF1,sF2,sF3,sF4,sF5,sF6,sF7,sF8,sF9:string;
  end;
 You are creating a new data type which is just the right thing for the needs of this program.

Now, in the private declarations, add.... 
rtParsed:TParsed;
function ParseRecord(sTmp:string):TParsed;
 ... just before the existing "function ReadFromByteFile(liStrt:longint):string;..."

Just after the "{$R *.DFM}" after the word "implementation" add.... 
{$R+}

function TDD42f1.ParseRecord(sTmp:string):TParsed;
begin
ParseRecord.sF1:='in func';
end;
 (The {$R+} has nothing to do with creating the new type, nor with ParseRecord, nor even with the {$R *.DFM}, but it is a good idea, and this was an easy moment to mention it!)

In the buDoItClick procedure, just after the existing "boDFInOpen:=false;", add... 
rtParsed:=ParseRecord(sTmp);
laTmp.caption:=rtParsed.sF1;
 ... (replacing the existing "laTmp.caption:=....")

Run the program again, click "Do it", and you should see "in func" appear in laTmp.

Add the following labels to the form. You don't have to do all of them if you see why not! laErr, laF1,laF2,laF3,laF4,laF5,laF6,laF7,laF8,laF9

Change ParseRecord so that it says.... 
function TDD42f1.ParseRecord(sTmp:string):TParsed;
begin
ParseRecord.sErr:='no error reporting yet';
ParseRecord.sF1:='field 1';
ParseRecord.sF2:='field 2'; (*No need to do the others yet*)
end;
 ... and in buDoItClick, replace "laTmp.caption:=rtParsed.sF1;" with.... 
laErr.caption:=rtParsed.sErr;
laF1.caption:=rtParsed.sF1;
laF2.caption:=rtParsed.sF2;
 Run the program again. Now clicking DoIt fills in some of the labels on the form, but only with the hard-coded text, so revise ParseRecord to be.... 
function TDD42f1.ParseRecord(sTmp:string):TParsed;
begin
ParseRecord.sErr:='no error';
ParseRecord.sF1:='';
ParseRecord.sF2:='';
ParseRecord.sF3:='';
ParseRecord.sF4:='';
ParseRecord.sF5:='';
ParseRecord.sF6:='';
ParseRecord.sF7:='';
ParseRecord.sF8:='';
ParseRecord.sF9:='';
if length(sTmp)<9 then begin
  ParseRecord.sErr:='String passed to ParseRecord was too short';
  end (*no ; here*)
 else begin
  ParseRecord.sF1:=copy(sTmp,1,10);
    sTmp:=copy(sTmp,12,length(sTmp)-12);
  ParseRecord.sF2:=sTmp;
  end;(*else*)
end;
 We are relying on the first field always consisting of 10 characters, and on it being followed by a single tab character or space. There is a single tab or space between the other fields in each record, too, and we could use that to break the string up. However, that might be tedious, so a different approach will be used. Crude, but simple! But vulnerable to changes in the data file format. (In fact, in the first edition of this program after the tutorial version was completed, this approach was abandoned. In the new, better version, the record is assumed to be field/tab/field/tab/field/tab, etc. It worked, and was more robust. Individual fields were validated against more strict format expectations.)

The next stage in programming this is to replace "ParseRecord.sF2:=sTmp;" with something that looks like a lot of typing. However, with good use of copy and paste, it isn't as bad as it looks. The "if length(sTmp)=42..." arises because my data has some fields in one format, others in the other. Sigh. (The field/tab/field/tab concept mentioned a moment ago made this clumsyness unnecessary.) That's the bad news.. well part of it. The good news is that I'm not going to bother gathering the data in the last three fields. 
  ParseRecord.sF2:=copy(sTmp,1,5);
    sTmp:=copy(sTmp,7,length(sTmp)-7);
  if length(sTmp)=42 then begin
      ParseRecord.sF3:=copy(sTmp,1,5);
        sTmp:=copy(sTmp,7,length(sTmp)-7);
      ParseRecord.sF4:=copy(sTmp,1,5);
        sTmp:=copy(sTmp,7,length(sTmp)-7);
      ParseRecord.sF5:=copy(sTmp,1,5);
        sTmp:=copy(sTmp,7,length(sTmp)-7);
      end (*no ; here*)
     else begin
      end;(*else*)
  end;(*else*)
end; (*Of ParseRecord*)
 ... and in buDoItClick, expand the obvious section thus.... 
laErr.caption:=rtParsed.sErr;
laF1.caption:=rtParsed.sF1;
laF2.caption:=rtParsed.sF2;
laF3.caption:=rtParsed.sF3;
laF4.caption:=rtParsed.sF4;
laF5.caption:=rtParsed.sF5;
laF6.caption:=rtParsed.sF6;
laF7.caption:=rtParsed.sF7;
laF8.caption:=rtParsed.sF8;
laF9.caption:=rtParsed.sF9;
 "What," you may be asking, "was the other bit of bad news?". The 6th field can be of varialble length. At this point in the program, we've nibbled off of sTmp all of the data we've moved to fields. Each time, so far, we've known how many bytes long the field will be. (In fact, this assumption proved wrong, but you can stick with it for the purposes of the tutorial.) Not so for the 6th field. We have to look for the next tab or space... and we're going to assume it is a tab... and chop the field off that way. Not too hard, actually... it just looks scary. Just before the "end; (*Of ParseRecord*)" in what we have above, add.... 
bTmp:=pos(chr(9),sTmp);(*chr(9) is "tab"*)
ParseRecord.sF6:=copy(sTmp,1,bTmp-1);

Going loopy

Onward! Now we're going to add the programming that will make the program search backwards through the data file a record at a time until either it finds the day it has been told to look for, or it finds that there is no record for that date in the file.

While we "have" the date in ParseRecord.sF1, it is held as a string... not good for comparing to a target date. For that matter, we don't (yet!) have a target date available to the program. Add declarations for the following variables, just after the declaration "rtParsed:TParsed;"... 
    dtTmp,dtTarget:TDateTime;
    bTarMo,bTarDa,bDOW:byte;
    liTarYr:longint;
 ... and add the following to buDoItClick, just after "rtParsed:=ParseRecord(sTmp);".... 
bTarMo:=10;
bTarDa:=31;
liTarYr:=2003;
dtTarget:=encodedate(liTarYr,bTarMo,bTarDa);
sTmp:=rtParsed.sF1;
dtTmp:=encodeDate(strtoint(copy(sTmp,7,4)),
  strtoint(copy(sTmp,1,2)),
  strtoint(copy(sTmp,4,2)));
if dtTmp=dtTarget then laTmp.caption:='Date in record matches target' (*no ; here*)
  else laTmp.caption:='Date in record does NOT match target';
 The above gives a nice example of starting small and building. The very crude entry of the target date will (obviously?) be replaced by more elegant and useful programming, but it serves at this stage to work while we develop other aspects of the program. Also, before the program is revised to loop through records to find the one we want, setting the target date will be moved elsewhere. We'll do that now.

Create a listbox on your form. Call it lbMonths. Populate the "items" stringlist with Jan, Feb, Mar, Apr, etc (one per line). Adjust the size of the control so that all twelve months show.

Create an OnCreate event handler for the form with the following in it... 
lbMonths.ItemIndex:=9;(*N.B. Set to 0 for January*)
 Change the "bTarMo:=10;" in buDoItClick to... 
bTarMo:=lbMonths.ItemIndex+1;
 (The "+1" is needed as "Jan" is the "zeroth" item in the list, Feb the "first", etc)

Move... 
bTarDa:=31;
liTarYr:=2003;
 from buDoItClick to the form's OnCreate handler, for now.

A Snag!

ARGHH!!! Did you notice the flaw in my design? I didn't... and it took a while to pin down. I'm happy to say that it isn't going to be a problem in the final version of the program.

As one always should, I was testing the program after this latest advance. Seemed to be working okay.... but once in a while I got a "EConvertError" message, "Invalid arguement to EncodeDate" when I clicked DoIt after changing the month for the target date. It seemed... at first... that odd numbered dates worked okay, even numbered dates not. June always misbehaved. Eventually I realised I was asking for "June 31st" to be converted. Easy enough to understand... when you finally hit the right question to ask!

We won't fix this problem for the moment. It won't exist in the final version of this.

Do you have a date?....

Next we'll turn to making the program search for the record for a particular date.

The first thing we need to do is to revise "function ReadFromByteFile( liStrt:longint):string;". Add "var" to the declaration as follows... 
function ReadFromByteFile(var liStrt:longint):string;
 ... and add var to the implementation header in the same place. Your program should still run after you add "var" in those two places.

Add boDone to the list of global boolean variables.

Revise buDoItOnClick as follows. Most of the ingredients come from what we had before, but they've been rearranged. 
procedure TDD42f1.buDoItClick(Sender: TObject);
begin
assignfile(dfIn,'DemoData/IBM.txt');
reset(dfIn);
boDFInOpen:=true;
liTmp:=filesize(dfIn)-1;
bTarMo:=lbMonths.ItemIndex+1;
dtTarget:=encodedate(liTarYr,bTarMo,bTarDa);
boDone:=false;
repeat
  sTmp:=ReadFromByteFile(liTmp);
  rtParsed:=ParseRecord(sTmp);
  if rtParsed.sErr='no error' then begin (*1*)
      sTmp:=rtParsed.sF1;
      dtTmp:=encodeDate(strtoint(copy(sTmp,7,4)),
      strtoint(copy(sTmp,1,2)),
      strtoint(copy(sTmp,4,2)));
      if dtTmp=dtTarget then begin (*2*)
          laTmp.caption:='Date in record matches target';
          boDone:=true;
          end (*2 no ; here*)
        else begin(*2*)
          laTmp.caption:='Record with that date not found in dataset.';
          if dtTmp'String passed to ParseRecord was too short'
        then boDone:=true;
      end; (*else 1*)
  if liTmp=-1 then boDone:=true;
until boDone;
closefile(dfIn);
boDFInOpen:=false;
laErr.caption:=rtParsed.sErr;
laF1.caption:=rtParsed.sF1;
laF2.caption:=rtParsed.sF2;
laF3.caption:=rtParsed.sF3;
laF4.caption:=rtParsed.sF4;
laF5.caption:=rtParsed.sF5;
laF6.caption:=rtParsed.sF6;
laF7.caption:=rtParsed.sF7;
laF8.caption:=rtParsed.sF8;
laF9.caption:=rtParsed.sF9;
end;
 I'm not convinced that the code above is the best I've ever written. I'm not even confident it will work in every case... but it works in a number of cases I've tried, so we'll move on again.

Scanning multiple files

The next stop is a way to scan all of the files in a given folder, fetching the price for each stock on the target date. Sometimes it is worth expressing your intentions in pseudocode. In this instance, we want something like.... 
set up any needed starting conditions
.. including:
....Get list of files
....Set "TooBigLastFile" to the number of  files in the folder
....Set "this file" to 0 ("This file" being a number for which file we've got to)
repeat
  check "this file" for datum
  if found then
        deal with datum
     else deal with failure
add 1 to "this file"
until "this file">="TooBigLastFile"
The code that will derive from the above will be the new buDoItClick handler. Before we start building that, we need to move all of what we currently have in the current buDoItClick handler to a new procedure called ProcessOneInputFile. Don't test what you are going to do next until you get down to "Now you can test the latest changes". To do what we need....

Add... 
procedure ProcessOneInputFile;
 in the private declarations just after "function ReadFromByteFile(var liStrt:longint):string;"

Change the header "procedure TDD42f1.buDoItClick(Sender: TObject);" to.... 
procedure TDD42f1.ProcessOneInputFile;
 ... and, just before "procedure TDD42f1.ProcessOneInputFile;", insert.... 
procedure TDD42f1.buDoItClick(Sender: TObject);
begin
ProcessOneInputFile
end;
 (It is a little dangerous to fool with the declarations and descriptions of event handlers "by hand" like this, but you can get away with what we've done if you're careful!) Now you can test the latest changes. the overall effect of the program will be the same, but the parts of the job are broken up differently.

Because it compartmentalises aspects of the work, which will be a big help if ever you go back to revise the code, we're going to make ProcessOneInputFile a function. You may well need to revise the code. The supplier of the data may change its format, for a start. The function is going to return a string which tells you there was no error within ProcessOneInputFile (or describes the error) and a number: the price for that company on the target date. As we did with ParseRecord, we are going to create a record (term used in slightly different sense) of our own to "wrap" the two data into a single "datum", so that we can have ProcessOneInputFile as a simple function. The alternative is to pass information from ProcessOneInputFile in global variables, which is a (widely used) Bad Idea which seems harmless enough at the time, but often makes tracking down bugs tedious.

Regarding the term "record": If crops up in this tutorial in two senses. Within a database, it means a row of a table. In the work we are doing, the files with prices for the different stocks are database tables. One line from any of those files is one "record" in the database sense.

In Delphi (and the underlying Pascal), the term "record" is used when declaring a new data type. We've already used it this way in this tutorial when we created the data type "TParsed". If you go to your Delphi sourcecode, put your insertion point somewhere in the word "record", do ctrl-F!, and select the entry from OBPASCAL.HLP, you can read more in the Object Pascal Language Reference.

So much for grand theory. To "just do it"... put the following into your program just before "type TParsed = record"... 
type
  TFromInput = record
      sErr:string;
      rValue:real;
  end;
 Just after "liTarYr:longint;", add.... 
rtFrmIP:TFromInput
 (Be sure that much still runs) Change "procedure ProcessOneInputFile;" to... 
function ProcessOneInputFile:TFromInput;
 ... and change "procedure TDD42f1.ProcessOneInputFile;" to... 
function TDD42f1.ProcessOneInputFile:TFromInput;
 ... and make buDoItClick be.... 
procedure TDD42f1.buDoItClick(Sender: TObject);
begin
rtFrmIP:=ProcessOneInputFile;
end;

See that still runs.

Just before the "end;" at the end of ProcessOneInputFile, add... 
ProcessOneInputFile.sErr:=rtParsed.sErr;
ProcessOneInputFile.rValue:=42.42;

See that still runs. One it does, add... 
liErr:longint;
rTmp:real;
 ... to the program's global variables, add two labels to the form... laFromFile1 laFromFile2

and replace the original two ProcessOneInputFile lines with.... 
ProcessOneInputFile.sErr:=rtParsed.sErr;
ProcessOneInputFile.rValue:=999.999;(*Or any other rogue value you want.
       Signals "not valid datum"*)
if rtParsed.sErr='no error' then begin
   val(rtParsed.sF3,rTmp,liErr);
   if liErr=0 then begin
     ProcessOneInputFile.rValue:=rTmp;
     end (*no ; here*)
    else begin
     ProcessOneInputFile.rValue:=liErr;
     ProcessOneInputFile.sErr:='Error found during convert to real. '+
       'Error has been passed back in ProcessOneInputFile.rValue';
     end(*else*)
   end;
 (N.B.: The "end" you see above is IN ADDITION to the function's final "end") ... and revise buDoItClick, making it.... 
procedure TDD42f1.buDoItClick(Sender: TObject);
begin
rtFrmIP:=ProcessOneInputFile;
laFromFile1.caption:=rtFromIP.sErr;
laFromFile2.caption:=floattostr(rtFromIP.rValue);

Where have we been?

Set out here as it has been, it may seem that the road to the final product is long and tortuous. Not really... compared to trying to write the whole thing in one gargantuan effort with all features added in a single push. What this tutorial illustrates, among other things, is incremental development. We're getting to our goal one step at a time. Try to take two steps, and you'll waste a lot of time getting to the bottom of which bit isn't working so that you can fix it. 
The simple life:
Repeat....
...Change one thing.
...Repeat
.........See if program still works.
........If not, try fixing the one thing you fiddled with
........Until what you've done works
...Until the whole job is done.

Where are we going?

That little homily aside, we return to the joys of getting the project completed. Previously, I presented the pseudocode for what needs to be done. We have already developed the essentials at the heart of that. Now we need to work up the details.

Almost everything before the repeat section of the pseudocode will be done outside of "DoIt".

The list of files will be held in a FileListBox, a useful built-in component. Put one on the form. (It can be found on the System tab of the components palette, third item.) Call it fbFiles. Set TabStop false. You might eventually want to set its visible property to false. The mask value can be set to *.txt for our needs, as all of the data files have the .txt extension.

Before we can collect the files, we need to designate which folder they are in. Declare a string variable sFilesPath. In the form's OnCreate give it an initial value. If your data files are in C:\Delphi\DD42\DemoData, then set sFilesPath equal to 'C:\Delphi\DD42\DemoData'. We can get clever later about being able to designate different folders via the ini file, or interactively.

Declare a variable called called liLimitToLoop. (We'll be using this to hold what was referred to as "TooBigLastFile" in the pseudocode. Just after the line setting sFilesPath, add.... 
liLimitToLoop:=fbFiles.items.count+1;

Now we're set to return to "DoIt" and build it up.

I'll continue the description in the same style as I've used so far, but don't bother trying to enter the new material as the description unfolds.... I'll give you the complete new DoIt in a moment.

For the moment, Put (* and *) around the three lines we have in that so far. Create a local variable called liLoopCounter. (This is for what was called "this file" in the pseudocode.) Create an approximation of the final program, using commented material to stand in for incomplete things, and showmessage statements to convey the program's progress.

That should bring you to.... 
procedure TDD42f1.buDoItClick(Sender: TObject);
var liLoopCounter:longint;
begin
liLoopCounter:=0;
repeat
  sTmp:=fbFiles.items[liLoopCounter];
  showmessage('Here will go check of file '+sTmp);
  (*rtFrmIP:=ProcessOneInputFile;
    laFromFile1.caption:=rtFromIP.sErr;
    laFromFile2.caption:=floattostr(rtFromIP.rValue);*)
  if true{datum found} then
        {deal with datum}
     else {deal with failure};
  inc(liLoopCounter);
until liLoopCounter>=liLimitToLoop;
end;

Now that we're scanning the right files, so we need to go back to ProcessOneInputFile and modify it to accept a filename when called. Add "(sFilePathAndName:string)" in the two places it is needed, replace 'DemoData/IBM.txt' within ProcessOneInputDevice with sFilePathAndName, and then be sure program will still run. It won't do anything interesting yet, but we can catch some typos.

Finally, make the changes in DoIt to create.... 
procedure TDD42f1.buDoItClick(Sender: TObject);
var liLoopCounter:longint;
    sCoName:string;
begin
liLoopCounter:=0;
repeat
  sTmp:=fbFiles.items[liLoopCounter];
  sCoName:=copy(sTmp,1,pos('.',sTmp)-1);
  showmessage('Here will go check of file '+sTmp);
  rtFrmIP:=ProcessOneInputFile(sFilesPath+'\'+sTmp);
  laFromFile1.caption:=rtFrmIP.sErr;
  laFromFile2.caption:=floattostr(rtFrmIP.rValue);
  if true{datum found} then
        {deal with datum}
     else {deal with failure};
  inc(liLoopCounter);
until liLoopCounter>=liLimitToLoop;
end;
 That will give us something that goes to the data files and fetches prices. Only a little more is now needed to process errors which might arise, and send the collected data to an output file. Make the lines before and including the "until ".... 
  if rtFrmIP.sErr='no error' then begin
        showmessage('On the date requested, '+sCoName+'''s close was '+
            floattostr(rtFrmIP.rValue));
            end (* no ; here*)
     else showmessage ('Error encountered when looking at '+sCoName);
  inc(liLoopCounter);
until liLoopCounter>=liLimitToLoop;
 You can take out the earlier "showmessage('Here will go check of file '+sTmp);" at this stage.

And finally...

All that's left to create a program that does the core job of what we set out to do is....

Open the output file. It will use the handle dfOut Put.... 
writeln(dfOut,sCoName+' '+floattostr(rtFrmIP.rValue));
 ...in place of "showmessage('On the date requested, '+sCoName+'''s close was '+floattostr(rtFrmIP.rValue));" Arrange to close dfOut in the relevant places.

To open the file crudely, just put the following into the early part of DoIt. It will cause existing DD42OP.txt files to be overwritten without warning.. Eventually, a default output filename will be fetched from the ini file, if a file exists of that name permission to overwrite will be sought, it will be possible to change the name. For now, the following works, if crudely... 
assignfile(dfOut,'C:\My Documents\DD42OP.txt');
rewrite(dfOut);
boDFOutOpen:=true;
 You should also put boDFOutOpen:=false; in the form's OnCreate handler.

To arrange the necessary closing, put... 
if boDFOutOpen=true then closefile (dfOut);
 in the FormClose procedure, and... 
if boDFOutOpen=true then closefile (dfOut);
boDFOutOpen:=false;
 ... just before the "end" at the end of "DoIt"

Concluding remarks

Well... there you have... something! As I've got further into this, it has proven more and more difficult to convey sensibly the changes and tweaks needed to evolve the program from the crude basic version into something with the smooth edges that a program you'd enjoy using every day would have.

I will do more work on this program later, and, maybe release the sourcecode of that more polished version... but there are a number of other projects clamouring for attention before then... not least getting this text onto the web for you! You'd be surprised how much work that entails, even once the basic text is done. (About 90 minutes, to date, as I work through the file. Much more to do.. the text editing is only part of the job.) In the meantime, I hope what IS in this tutorial was worth the time you spent working through it!

When the program goes from delvelopment to real-life use, the date for which data is sought will default to the first non-weekend day before the current day. The built in functions "now" and "dayofweek" will make the programming quite simple.
   Search this site or the web        powered by FreeFind
 
  Site search Web search
Site Map    What's New    Search The search engine is not intelligent. It merely seeks the words you specify. It will not do anything sensible with "What does the 'could not compile' error mean?" It will just return references to pages with "what", "does", "could", "not".... etc.


Click here if you're feeling kind! (Promotes my site via "Top100Borland")
Ad from page's editor: Yes.. I do enjoy compiling these things for you... hope they are helpful. However.. this doesn't pay my bills!!! Sheepdog Software (tm) is supposed to help do that, so if you found this stuff useful, (and you run an MS-DOS or Windows pc) please visit my freeware and shareware page, download something, and circulate it for me? Links on your page to this page would also be appreciated!

Click here to visit editor's freeware, shareware page.

Link to Tutorials main page
How to email or write this page's editor, Tom Boyd

Valid HTML 4.01 Transitional Page WILL BE tested for compliance with INDUSTRY (not MS-only) standards, using the free, publicly accessible validator at validator.w3.org


If this page causes a script to run, why? Because of things like Google panels, and the code for the search button. Why do I mention scripts? Be sure you know all you need to about spyware.

....... P a g e . . . E n d s .....