Sorting data in fixed file format

[booksnore]

I'm looking for a way to sort text files consisting of fixed file
format. The files are big, typically over 10 million records and they
consist of about 100 fields with the record being over 600 bytes in
length. I need to
sort on a combination of 8 of the fields. Has anyobe attempted a to sort
data in this way using C#? Performance and scalability are the main
factors - the input data I
have is likely to grow probably up to 40 million records. I want to
avoid loading data into a database just to sort it.

Joe

*** Sent via Developersdex http://www.developersdex.com ***

[Ignacio Machin \( .NET/ C# MVP \)]

Re: Sorting data in fixed file format

Hi,

IMO your best (if not unique) solution is to put everything in a database,
sort it and then export it. With this amount of data there is no a better
solution.

Importing/Exporting is very easy using DTS packages. let me know if you
need code for it


--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"booksnore" <sattwell@pland atamgmt.com> wrote in message
news:OoG8$zmAGH A.2656@tk2msftn gp13.phx.gbl...[color=blue]
> I'm looking for a way to sort text files consisting of fixed file
> format. The files are big, typically over 10 million records and they
> consist of about 100 fields with the record being over 600 bytes in
> length. I need to
> sort on a combination of 8 of the fields. Has anyobe attempted a to sort
> data in this way using C#? Performance and scalability are the main
> factors - the input data I
> have is likely to grow probably up to 40 million records. I want to
> avoid loading data into a database just to sort it.
>
> Joe
>
> *** Sent via Developersdex http://www.developersdex.com ***[/color]

[Kevin Spencer]

Re: Sorting data in fixed file format

You have over 10 million records in a fixed file format, and you don't want
to put it into a database? Dude, that's what databases are *for*. Fixed file
formats are fine for storing static data. But filtering, selecting,
ordering, that's all stuff for databases.

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
You can lead a fish to a bicycle,
but it takes a very long time,
and the bicycle has to *want* to change.

"booksnore" <sattwell@pland atamgmt.com> wrote in message
news:OoG8$zmAGH A.2656@tk2msftn gp13.phx.gbl...[color=blue]
> I'm looking for a way to sort text files consisting of fixed file
> format. The files are big, typically over 10 million records and they
> consist of about 100 fields with the record being over 600 bytes in
> length. I need to
> sort on a combination of 8 of the fields. Has anyobe attempted a to sort
> data in this way using C#? Performance and scalability are the main
> factors - the input data I
> have is likely to grow probably up to 40 million records. I want to
> avoid loading data into a database just to sort it.
>
> Joe
>
> *** Sent via Developersdex http://www.developersdex.com ***[/color]

[Joe]

Re: Sorting data in fixed file format

Thanks Ignacio, what sort of code is it that you have? I would be
interested in taking a look.
regards
Joe

Joe

--
Sent via .NET Newsgroups

http://www.dotnetnewsgroups.com

[Ignacio Machin \( .NET/ C# MVP \)]

Re: Sorting data in fixed file format

Hi,

this is what I use, I create a DTS from enterprise manager , where the data
is comnig from and where to put it, then select "create a file" or something
similar in the DTS wizard, it does create a .dts file this is the one you
will use later

Here is the code, note that I change the datasource from the code, you can
do a similar thing with the destination:
Also you need to add a reference to DTS COM library

using DTS;
using System.Data;
using System.Data.Sql Client;



void RunPackage( string packSource, string packName, string dataSource)
{
try
{
Package2Class package = new Package2Class() ;
object pVarPersistStgO fHost = null;

// if you need to load from file
package.LoadFro mStorageFile(
packSource,
null,
null,
null,
packName,
ref pVarPersistStgO fHost);

/*
package.LoadFro mSQLServer(
"",
null,
null,
DTSSQLServerSto rageFlags.DTSSQ LStgFlag_UseTru stedConnection,
null,
null,
null,
"Test Import Package",
ref pVarPersistStgO fHost);
*/

package._Packag e_Connections.I tem(1).DataSour ce = dataSource;
package.Execute ();
package.UnIniti alize();

// force Release() on COM object
//
System.Runtime. InteropServices .Marshal.Releas eComObject(pack age);
package = null;
}
catch(System.Ru ntime.InteropSe rvices.COMExcep tion e)
{
Console.WriteLi ne("COMExceptio n {0}", e.ErrorCode.ToS tring() );
Console.WriteLi ne("{0}", e.Message);
Console.WriteLi ne("{0}", e.Source);
Console.WriteLi ne("Stack dump\n{0}\n", e.StackTrace);
Console.ReadLin e();
}
catch(System.Ex ception e)
{
Console.WriteLi ne("Exception") ;
Console.WriteLi ne("{0}", e.Message);
Console.WriteLi ne("{0}", e.Source);
Console.WriteLi ne("Stack dump\n{0}\n", e.StackTrace);

Console.ReadLin e();
}
}



--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation



"Joe" <booksnore@nets cape.net> wrote in message
news:uw9GR$mAGH A.1288@TK2MSFTN GP09.phx.gbl...[color=blue]
> Thanks Ignacio, what sort of code is it that you have? I would be
> interested in taking a look.
> regards
> Joe
>
> Joe
>
> --
> Sent via .NET Newsgroups
> http://www.dotnetnewsgroups.com[/color]

[booksnore]

Re: Sorting data in fixed file format

I want to sort the data in the file - that's it nothing else. Unix has a
sort command, I wondered if there was an equivalent in Windows.

Joe

*** Sent via Developersdex http://www.developersdex.com ***