general advice on data storage

[Mike]

Hi,

I'm new to vb.net and would be grateful if anyone could offer some
advice on the best way to approach this problem:

I wish to dynamically capture pricing information e.g. 10 stock prices
and use this data to generate a number of graphs over time (in real
time) of each share price. This means for each share I need to store a
price and a timestamp.

I would need to collect around 600 pairs of data per stock (once per
second for 10 mins), so around 6000 in total, then roll the data
forward so that I only ever keep 600 records i.e. always have the last
10 minutes worth of data for each stock.

I would therefore like to know the best method to store this data on
the fly. I can only think of an array or dataset as a way to do this
but maybe there are better solutions I am unaware of?

Thanks for any help,

Mike

[Cor Ligthert [MVP]]

Re: general advice on data storage

Mike,

It is a very rough answer for a newsgroup, however I think that I would
first try for an SQLExpress or the old MSDE database. With a table per
share.

Than add it everytime rows using the insert to the share tables with a new
datatime field as key.

Creating those tables is easy to do in VBNet in a for each loop.

To get than a datatable for the last 10 minutes is just getting that with a
fill with a "where" clause the < datatime. It is not slow however I don't
know if it is quick enough.

It needs of course consequent deleting with another seperate running program
from the old rows.

Using the dataset/datatable itself inside a dataset will almost for sure
break you up in VS2003. Deleting of that will probably be to slow (this
should be improved in VS2005, however I did not try it yet)

Just as first idea when I saw your question.

Cor

[Mike]

Re: general advice on data storage

Thanks Cor - much appreciated. I'll try out your suggestions.

[Jevon]

Re: general advice on data storage

How often will the graph be redrawn?

In addition to Cor's reply, here's another option:
Use a linked list. Something like the following [psuedocode]:

public struct StockHistoryIte m
Dim price as double
Dim timestamp as DateTime
end struct

class StockHistory
Private data() as StockHistoryIte m
Private startData as Long = 0
Private endData as Long = 0
Private maxSize as Long = 0

public Sub New(byref size as Long)
if Size>0 then
maxSize = size
Redim data(maxSize) As StockHistoryIte m
else
throw new Exception - invalid
end if
end sub

public sub AddItem(ByVal newItem as StockHistoryIte m)
if endData = maxSize then
endData = 0
else
endData += 1
end if
data(endData) = newItem
if endData = maxSize Then
startData = 0
else
startData += 1
end if
end sub
end class

Bear in mind I don't think the upper bound stuff is correct - you might need
(endData + 1) in the "If" statements involving it. You obviously need
functions to retrieve data - you could implement IEnumerable or have
something like:
public sub GetItem(byref index as Long)
if index >=0 AND index <= maxsize - 1 then
' You also need to check in case you haven't yet filled the
structure with values.
GetItem = data(Dim item As Long = (index + startData) Mod
maxSize)
end if
end sub


If you took Cor's route, you don't necessarily need an entirely separate
program, you could clean up old rows each time you draw the graph.

Jevon


"Mike" <mykyp@pearcey2 001.freeserve.c o.uk> wrote in message
news:1130762090 .933423.301490@ g14g2000cwa.goo glegroups.com.. .[color=blue]
> Hi,
>
> I'm new to vb.net and would be grateful if anyone could offer some
> advice on the best way to approach this problem:
>
> I wish to dynamically capture pricing information e.g. 10 stock prices
> and use this data to generate a number of graphs over time (in real
> time) of each share price. This means for each share I need to store a
> price and a timestamp.
>
> I would need to collect around 600 pairs of data per stock (once per
> second for 10 mins), so around 6000 in total, then roll the data
> forward so that I only ever keep 600 records i.e. always have the last
> 10 minutes worth of data for each stock.
>
> I would therefore like to know the best method to store this data on
> the fly. I can only think of an array or dataset as a way to do this
> but maybe there are better solutions I am unaware of?
>
> Thanks for any help,
>
> Mike
>[/color]

[Jevon]

Re: general advice on data storage

Sorry, I should add to this that it obviously isn't necessarily the best
option if memory is an issue, but for the amount of data you mention it
shouldn't be too much of a problem.

Jevon

"Jevon" <please@ask.com > wrote in message
news:ekOKuYk3FH A.1148@tk2msftn gp13.phx.gbl...[color=blue]
> How often will the graph be redrawn?
>
> In addition to Cor's reply, here's another option:
> Use a linked list. Something like the following [psuedocode]:
>
> public struct StockHistoryIte m
> Dim price as double
> Dim timestamp as DateTime
> end struct
>
> class StockHistory
> Private data() as StockHistoryIte m
> Private startData as Long = 0
> Private endData as Long = 0
> Private maxSize as Long = 0
>
> public Sub New(byref size as Long)
> if Size>0 then
> maxSize = size
> Redim data(maxSize) As StockHistoryIte m
> else
> throw new Exception - invalid
> end if
> end sub
>
> public sub AddItem(ByVal newItem as StockHistoryIte m)
> if endData = maxSize then
> endData = 0
> else
> endData += 1
> end if
> data(endData) = newItem
> if endData = maxSize Then
> startData = 0
> else
> startData += 1
> end if
> end sub
> end class
>
> Bear in mind I don't think the upper bound stuff is correct - you might
> need (endData + 1) in the "If" statements involving it. You obviously need
> functions to retrieve data - you could implement IEnumerable or have
> something like:
> public sub GetItem(byref index as Long)
> if index >=0 AND index <= maxsize - 1 then
> ' You also need to check in case you haven't yet filled the
> structure with values.
> GetItem = data(Dim item As Long = (index + startData) Mod
> maxSize)
> end if
> end sub
>
>
> If you took Cor's route, you don't necessarily need an entirely separate
> program, you could clean up old rows each time you draw the graph.
>
> Jevon
>
>
> "Mike" <mykyp@pearcey2 001.freeserve.c o.uk> wrote in message
> news:1130762090 .933423.301490@ g14g2000cwa.goo glegroups.com.. .[color=green]
>> Hi,
>>
>> I'm new to vb.net and would be grateful if anyone could offer some
>> advice on the best way to approach this problem:
>>
>> I wish to dynamically capture pricing information e.g. 10 stock prices
>> and use this data to generate a number of graphs over time (in real
>> time) of each share price. This means for each share I need to store a
>> price and a timestamp.
>>
>> I would need to collect around 600 pairs of data per stock (once per
>> second for 10 mins), so around 6000 in total, then roll the data
>> forward so that I only ever keep 600 records i.e. always have the last
>> 10 minutes worth of data for each stock.
>>
>> I would therefore like to know the best method to store this data on
>> the fly. I can only think of an array or dataset as a way to do this
>> but maybe there are better solutions I am unaware of?
>>
>> Thanks for any help,
>>
>> Mike
>>[/color]
>
>[/color]

[Mike]

Re: general advice on data storage

Thanks for the alternative. I would ideally like to redraw the graph
each time a new data point is stored, so I'm hoping once a second.

I'm not sure whether the performance of graphing add-ins would be able
to handle this or not, but maybe I could resort to drawing it if they
can't. My ultimate aim would be to have each of these graphs running
within a datagrid cell (which would most likely require me to draw
manually) but that is way beyond my current level so I'll save that
part of it as a project for the future :-)

Thanks again,

Mike