Introduction To Data Files

Data Files

Why Do You Need To Know About Data Files?
Facts About Data Files
Creating And Using Flat Files In C
Creating And Using Flat Files in Visual Basic
Problems

An Introduction To Data Files

If you work with a computer you work with files. Sometimes those files contain information about something you are writing - in a word processor, for example. Sometimes those files contain other information like results of calculations. Sometimes those files contain results of measurements you took in a laboratory or out in the field.

In this lesson we are going to examine data files. There are numerous good reasons why you need to understand some basic ideas about data files.

The amount of data you can store in a data file on a disk is determined by the precision of the measurements you take and the number of data points you want to record.
You may want to write programs that take and store data, and you need to understand a little bit about file structure when you do that.

In both cases you need to know something about file structure and some details of the characteristics of data files. That's what this lesson is about.

Let's look at the life history of a typical data file.

You use some instrument to measure data - voltage, temperature, whatever.

The instrument manufacturer supplies a utility program that lets you store the data in a file. Alternatively, you write a program that controls an instrument and you write the program to store data in the file.

Later you want to analyze the data and you load your file into your favorite analysis program - Mathcad, Matlab, Excel, whatever.

Through all of this, you need to know something about data files. That's what this lesson is about.

Goals For This Lesson

It's pretty clear that data files are useful. We've come a long way from the days when we wrote data into a lab notebook for storage or kept a pile of computer printouts. So what do you need today?

Given a need to store data in a file,
Be able to use data files in general application programs.
Be able to write C or Visual Basic programs that will write data within a program into a file.
Be able to explain the structure of a file you created
in C or Visual Basic.
Be able to determine file size as a function of measurement
precision and number of data points.

Data Files

In this section we are going to discuss what a flat file looks like. Later sections will show specific program functions that will let you open and close flat data files and write data to the files you create. First we will examine how files are built up. Then we will look at details of creating files and writing data to files in some popular programs.

In order to manipulate data as described earlier you will need to understand some basic facts about how files can be constructed. In particular, you'll need to know the following.

Data files stored as text files (a txt extension in DOS/Windows systems) are stored as a sequence of characters.
Every character is stored as a single byte of data. You know that a byte can store a number from 0 to 255. Every character on the keyboard has a numerical representation.
There is a standard code for character representations. That code is the American Standard Code for Information Interchange, that is ASCII, and that is pronounced "Ask-ee". When you strike a key on the keyboard, the ASCII code for that key is what is transmitted to your computer.

Maybe the most important item on the list above is that every character is stored in a byte in a file. If you have that concept, then you can compute how much information can be stored on a disk.

Let's just take a single megabyte (1MB). That's one million (1,000,000) bytes.
Disks can store more than that. Floppies can store 1.44 MB and hard drives can store many gigabytes (A gigabyte is one billion bytes.)
If we can figure out how much you can get in a megabyte you can figure out how much you can get on a floppy or a hard disk.

As this is written, I'm reading a book.

By my count it had about 2500 characters on a random page I picked.
That means it would take 2500 bytes to store the text on a single page.
By that count, one megabyte could store 400 pages (2500 bytes/page x 400 pages = 1,000,000 bytes)
The particular book I was reading has only 267 pages, so the entire book could fit on a single floppy disk.

You can store a large amount of data even on a single floppy disk. Now, there are higher density disks that hold 100 megabytes or 250 megabytes, so consider these problems.

EXAMPLE

How many 267 page books will fit on a 100 megabye disk?

Assume 2500 characters per page, or 2500 bytes per page.
2500 x 267 = 667,500 bytes/book 0r .6675 megabytes/book
Therefore, the number of books is 100/.6675 = 149.8. Let's call it 149 books, or even 150.
That's a lot easier than carrying the hard copy version in a backpack.

To get started we will create a simple data file. We will start with a small file first using the data below

1.5	34.451
3	33.779
4.5	33.152

Open a simple text editor. (If you are in Windows, that will be Notepad. In Unix it might be emacs.) Then do the following, and DO NOT type any extra characters!

Type a "1" a "." (that's a period) and a "5".
Type a tab. This inserts a tab character between the "5" and the next set of numbers.Type the next set of numbers, i.e. "34.451".
Hit Enter. (You really are typing a carriage return and a line feed.)
Continue with all lines.
Save the file.

The file that you have created is a simple data file but it has several interesting properties.

The file can be loaded by many different applications. The reason many applications can load one file is that a simple file structure of this type is used by many different applications.

The file structure is so simple that you pretty much know every chacter in the file. They are the characters you typed.

Word processors can load this file. But if you save it as a word processor file you will save information about formatting that is not contained in this file.
Spreadsheets can load this file. But if you apply functions to your spreadsheet you are adding information beyond what is contained in this file.

Let's examine exactly what those characters were that you typed into the file. Here's the sequence of characters for the first line or so that you typed. Note that when you hit the Enter key you are actually typing two characters.

What is typed	What you get	ASCII Character
1	1	49
.	.	46
5	5	53
TAB	TAB	09
3	3	51
4	4	52
.	.	46
4	4	52
5	5	53
1	1	49
ENTER	CR	13
	LF	10

Every line on the list above has an ASCII character. (Click here to see a more complete list of ASCII characters.) That includes the tab, carriage return and linefeed characters. If you want to explore this further you might want to check out your favorite word processor. Many word processors have a feature that allows you to make the non-printing characters (like tabs, etc.) visible. Check that out and make sure that you can see how each of these characters shows up. (You might find the carriage return and line feed lumped into a single character shown with a paragraph mark.)

Every letter, every number, every punctuation mark has a specific numerical representation.
ASCII characters are what you manipulate when you put data into a text file.

Here are a few things you can do after you have copied the data and gotten it onto the computer clipboard.

Start a word processor on your computer and paste the data into the word processor.
Start a spreadsheet and paste the data into the spreadsheet.

Putting data on a clipboard and pasting it into an application is one way to get data into a program. However, in some cases you may need to do something else to get the data into a program. Analysis programs like Mathcad and Matlab can also load this data, but each program has a special way of getting the data into the workspace so it can be analyzed.

We need to examine the file contents of the simple file in more detail than we have to this point. Here is what is important.

When you type you generate characters. When you type a ""3"" you generate a character that is put into the editor for later storage in the file.
Even when you use the ""Enter"" key you are typing a character or two. You are entering a carriage return and/or a new line character, and each has its' own ASCII representation.
When you store the data in a file, you are storing characters - the characters that you typed.

Tabs, carriage returns and line feeds and other characters are ASCII characters, and you can write those characters into files from all popular programming languages. Also you can write strings of characters in other contexts. Here are some programming segments that write a string ""TEM"" followed by a carriage return and line feed. In these examples, a function is used to write the string to an instrument to cause it to measure a temperature.

In BASIC
C3$="TEM"+CHR$(&HD)+CHR$(&HA)
CALL IBWRT(DAU%,C3$)
In C/C++
long dau;
ibwrt(dau, "TEM\r\n", 5);

Some special characters are listed below.

ASCII Character Number	Name	BASIC form	C form
9	Tab	CHR$(&H9)	\t
10	LF	CHR$(&HA)	\n
13	CR	CHR$(&HD)	\r

The letters also have a numerical code. That's the first column in the table above. Every character - all 256 possible ASCII characters - correspond to one of the numbers fomr 0 to 255. That numerical correspondence allows us to identify each character with a number.

At this point you have the knowledge you need to understand how data can be stored in files. This isn't the entire story. Storing data as characters is often done, but it is also possible to store data in numerical formats. That's another topic. However, IEEE-488 instruments often transmit data using a character format, and that's something you can learn about in another lesson.

Creating And Using Flat Files in C

C has numerous functions for creation and deletion of files, as well as functions that open your files, write data to them and close them.

In this section we'll give you some functions we have written that allow you to create your own data files in a flat file format. You can embed these functions in programs that read laboratory data, and you can use these functions to create files with lab data that you can open in spreadsheets and analysis programs. With that capability you can then plot and analyze your lab data.

The functions we will give you will be the simplest possible. You can then modify these functions to "personalize""them for your own purposes.

The sequence of operations in a typical program might be the following.

Open a data file so that you can write data to it.
Write data to the file you have open.
Close the data file.

Here's a function that will open a file named DATAFILE.TXT on the A drive. If the file doesn't already exist, fopen creates it automatically. The function, fopen, used here is a standard, ANSI C, function. You can select the text and use Ctrl+C to copy the function if you are in a Windows system.

FILE *file; /* Declare a pointer to a FILE variable */
   /* Defined as a global to be available to */
   /* other functions   */
void OpenDataFile()
{
/* Subroutine to open a data file named "DATAFILE.TXT" in the same directory as the executable file.*/
char file_name[20] = "DATAFILE.TXT";
file = fopen (file_name, "w+");
     /* open data file "datafile.txt"    */
}

NOTE: The "w+" opens the file to write data to.

Here's a function that will write data to the file you opened. Again, fprintf is a standard ANSI C function. WriteDataToFile writes one data point, a tab character, the next data point and a carriage return/line feed.

void WriteDataToFile(float x_data, float y_data)
{
/* Subroutine to print values into the opened file. "present_time" is     */
/*      printed in one column, and "temperature" is printed in another.    */
/*      "present_time" and "temperature" must be sent down from function   */
/*      main.                                                              */
fprintf(file, "%3f\t", x_data); /* write x_data to data file */
fprintf(file, "%.3f\n", y_data); /* write y_data to data file */
}

After you are finished writing all the data to the file, you need to close the file. Here's a function for that.

void CloseDataFile()
{
/* Subroutine to close the data file named "DATAFILE.TXT" on the "A:" drive */
fclose (file); /* close data file "a:datafile.txt" */
}

That's all there is to it. These three functions - although somewhat bare bones - will suffice to get a data file for you. The precautions you need to take are:

Remember to keep the FILE pointer - *file - outside of all functions so that it is available to all the functions.
If you use this function twice (if you run your program twice, for example) the file will be written over, and you will lose your first data set. To get around this, rename the first file immediately.

Creating And Using Flat Files in Visual Basic

Visual Basic has numerous functions for creation and deletion of files, as well as functions that open your files, write data to them and close them.

The functions we will give you will be the simplest possible. You can then modify these functions to "personalize" them for your own purposes.

The sequence of operations in a typical program might be the following.

Open a data file so that you can write data to it.
Write data to the file you have open.
Close the data file.

First, you need to be able to open a file. That's simple in Visual Basic. Here's the code.

Open "DATAFILE.TXT For Output As #99

The Open statement will open a file with the name specified immediately after the word "Open". Normally you will open the file for Output, and you can assign it a number - to be used later in Print and/or Write statements. (Here we used #99.) You can also specify complete path information if the file will be located in another directory other than the one your programming is running in.

After you've opened the file, write data to it using something like this.

Print #99, Xdata, Ydata

When you're done writing data to your file, close it. Here's the code.

Close #99

There are better ways to specify a data file name. You can create A file I/O box using the standard FileListBox, DirListBox and DriveListBox tools. If you're good at Visual Basic, take a shot at that.

Problems