Using SQl to store aspx pages and memory problems

[Klaus H. Probst]

Re: Using SQl to store aspx pages and memory problems

The following suggestion may or may not work for you, but I'll chime in.

First off, to the folks that called the approach your existing app uses
"idiotic", I've seen classic ASP applications design this way that work
extremely well, are scalable and extremely manageable. However, the devil is
in the details. Just putting "pages" in IMAGE or TEXT fields on a SQL
database isn't some sort of magic pixie dust that solves all problems. It
takes a lot of thought and experimentation to get just right, and the
rewards in most cases offset the negative aspects of these designs.

Having said that, your approach is, as you've figured out by now, wrong.
Others have already mentioned the perf hit you're taking by forcing ASP.NET
to function in a way it wasn't designed to. But there are ways to get around
the problem.

First off, I'll assume that you have more content than code. That is, *most*
of your existing ASP/ASPX pages in the database contain stuff, like invoices
and whatnot, that do not require intervention by the parser to load and
display. If this is not the case then stop reading now =)

There are a lot of web applications nowadays that use their databases as
primary content storage repositories. Blogs are a good example of this.
However, they do not store code in the database, they store content. If you
can adapt to this type of scenario then all you have to do is create a sort
of "master page" or maybe an IHttpHandler implementation that looks at the
request (even a path), pulls the *content* from the database and then
displays it on a sort of templated page container. Think of a site like
MSDN. Lots of articles, yet all the pages look the same. The content is what
varies. The trick is to simply treat your "pages" as opaque chunks of
content, and "stream" them into a placeholder page that is rendered the same
way every time. You can pretty much do anything you want once you're hooked
into the ASP.NET processing pipeline.

If you do this you'll get rid of your perf problem and you'll still have the
flexibility and manageability of the database storage. This is web
application design 101 - always separate content from function and
structure.

Of course getting from point A to point B could be tricky, but maybe this
gives you some ideas =)

--
Klaus H. Probst, MVP

http://www.simulplex.net/

"matvdl" <matvdl@discuss ions.microsoft. com> wrote in message
news:DF375EB0-8373-4D91-B7F3-0400A8AE22E9@mi crosoft.com...[color=blue]
> DLM,
>
> Thanks for your response.
>
> I don't believe that SQL is my problem - the time it takes to do the[/color]
remote[color=blue]
> calls to the external program has not presented any real limitations or
> caused any significant time-delays. The problem that I have is trying to
> deal with the quanitity of files that I have and the combiling of those[/color]
pages[color=blue]
> once they have been returned. Any time issues that I have had are the[/color]
result[color=blue]
> of parsing and compilling the page.
>
> I understand your point - that asp.net was designed to have the pages
> pre-compiled before requests where made for the pages - I guess this is[/color]
one[color=blue]
> of the fundamental differances between asp and asp.net. From experiance[/color]
the[color=blue]
> design that has been implemented had no significant disadvantages when
> working with asp - in fact I believe that it had some very significant
> advantages - mainly due to the unlimited number of files it could manage.
>
> Question - is it practicale to have 10,000 pages in a single asp.net
> application. I wouldn't have thought so?
>
> I have no alternative but to deal with this many files as it is a legal
> requirment of the industry I work in is to be able to provide the original
> copies of invoices to clients - I can't go back to the original data and
> attempt to re-produce the same invoice just in case the data has changed.
>
> So - considering this is what I am stuck with - my problem mainly relates[/color]
to[color=blue]
> the memory increases - the slowness of the request could be dealt with by[/color]
the[color=blue]
> more grunt - but the ever increasing memory is more difficult. I believe
> that this is the result of the creation of a new class each time a new[/color]
page[color=blue]
> is displayed - although this page gets un-loaded once the info is[/color]
returned -[color=blue]
> are the increases in memory the result of the new class definition in the
> asp.net application?
>
> If this is the case - is it possible to remove this class definition once[/color]
it[color=blue]
> has been created? I have tried to delete the file - but this does not[/color]
appear[color=blue]
> to fix the problem.
>
> Does any of this make sense.
>
>
> --
> matthew
>
>
> "DLM" wrote:
>[color=green]
> > Go to this link:
> > http://www.theserverside.net/books/i...ngPerf/pag.tss
> >
> > We all know that there is an initial performance 'hit' the first time a[/color][/color]
ASP[color=blue][color=green]
> > .NET Page is requested, and this is true with the page sitting in it's
> > virtual directory on IIS. What you seem to be doing is increasing the[/color][/color]
round[color=blue][color=green]
> > trip time for any given request/response for a page to include the time[/color][/color]
it[color=blue][color=green]
> > takes to get the page from SQL Server. And this added time is database[/color][/color]
access[color=blue][color=green]
> > time which is probably longer than any HTTP request.
> >
> > I have never seen this kind of architecture before. The whole idea is to
> > serve pages from a highly scalable architecture such as IIS from the[/color][/color]
very[color=blue][color=green]
> > start, and not make the ASP .NET worker process wait for the files[/color][/color]
before it[color=blue][color=green]
> > can dynamically compile pages. In a sense you are using SQL Server as a[/color][/color]
web[color=blue][color=green]
> > server to your web server, which is highly unusual. You need to find a[/color][/color]
way to[color=blue][color=green]
> > place all your pages on the web server and use SQL Server only for data
> > retrieval/update.
> >
> > TAKE A LOOK AT MICROSOFT PATTERNS & PRACTICES
> >
> >[/color][/color]

[DancesWithBamboo]

RE: Using SQl to store aspx pages and memory problems

To understand why the solution you had working in clasic asp doesn't perform
in .net, you have to understand the internals of how these systems work.

In asp, every page was intrepreted runtime. It looked for your <% %> tags
and intrepeted the code between them real time as it ran each page. If your
page didn't caontain much server-side code and was mostly content then there
wasn't much to do. Basically the system was designed to look at each page
each time it was hit as necessary. This is why almost all systems (except
your) are faster in .net.

..net on the other hand "compiles" every page into IL code when the page
changes. This IL code is then intrepeted real time by the runtime engine.
Compiling is a lot of work. It outputs new code, checks syntax, checks type
casts, checks variable declarations, function calls, and another million
things. asp would just crap out if it didn't like what it saw. The key here
is that .aspx files are compiled once for every folder and every folder every
time a file changes in it. On a normal system, you take a hit once for each
folder that has a page in it requested. Then nothing changes for a long time
so the compiler may not do anything for months.

In your system, every time you push a page out from SQL to disk you are
instructing the compiler that there is a new page that it must compile (not
to mention the file system write which could possibly be the slowest thing
you could do on a Windows box). Now if you start loading all 10000 pages
into the same folder, the work it has to do will get worse as the more pages
are requested since there will be more pages to compile.

The key to making your system work is to do some major rearchitecting or
leave it in clasic asp. Why do you want to change? If you rearchitect, you
need to only store "data" in the database and keep code in files on the
server compiled. Then you load the pages with data from the db. This is how
content management systems work.

An example would be your invoice pages. I assume that many of your "pages"
in the database are invoices. You need to just store them as text along with
whatever data you need to make the page in SQL Server (table called Invoice?)
and then have a page called invoice.aspx on the server that uses code to grab
the data and format it for display(includi ng your graph control).

It really has never been a "good" design to put code in a database. It just
happens that classic asp was stupid enough that you could get away with it.
You need to leverage the systems for what they are good at; IIS/.net for page
serving and SQL server for data storage.

[matvdl]

Re: Using SQl to store aspx pages and memory problems

Klaus,

Thankyou for your help - I must admit that at the beginning I was a little
taken back by people's negative responses to the problem - I believe that
people were far to quick to make judgement without bothering to understand
the problem - I mentioned - I think in almost every response I mentioned that
I have around 10,000 pages - and no one bothered to provide much help with
the specific problem - I think the method I have used is not too bad - just
needs to be modified to work efficiently with asp.net.

Yes - most of the pages have mostly content and there is not allot of code
in each page. So I am trying to work out how to implement something similar
to your suggestion - but still provide the flexibility of enabling server
side code where needed.

What can make this is a little difficult is my lack of experience in asp.net
- but hopefully that will improve.

The solution I was looking at would be to create some specific server side
asp.net controls for some of the more specific tasks that I have - for
instance - I need to load charts at the server and therefore need to be able
to provide the data to these charts - sometimes this could be allot of data.
I want the data to be saved in the page so I thought that a asp.net control
could be the way to go - I could pass the data to the control and this
control would already be defined on the server.

So - if I have a string of HTML text with a embedded asp.net server control
and I use the response.write function will asp.net recognize that there is a
control with in the document and load it accordingly??

So there wouldn't be any <% %> within the html - just the server controls.

If this don't work that only way that I can image doing it would be to
implement my own method of tags - search from them at run-time and extract
the code between them and use the Eval function to evaluate the code. Does
not seem like a good solution - but I think it would work.

Part of my aim is to limit the dependence of the need to store data - once
the invoice or file is produced ideally I need future requests to only look
at this one file for it to display correctly - therefore the reasoning on
imbedding the data within the file - but that does not mean that the code
could be pulled out of the file and placed elsewhere. Having as few
dependencies as possible on what is required to display the page was part of
the original design and one that ensures that future system changes will not
inadvertently make viewing older invoices fail. Something that has worked
well over the past 5 years.

Are these the sorts of ideas you were thinking off??


--
matthew


"Klaus H. Probst" wrote:
[color=blue]
> The following suggestion may or may not work for you, but I'll chime in.
>
> First off, to the folks that called the approach your existing app uses
> "idiotic", I've seen classic ASP applications design this way that work
> extremely well, are scalable and extremely manageable. However, the devil is
> in the details. Just putting "pages" in IMAGE or TEXT fields on a SQL
> database isn't some sort of magic pixie dust that solves all problems. It
> takes a lot of thought and experimentation to get just right, and the
> rewards in most cases offset the negative aspects of these designs.
>
> Having said that, your approach is, as you've figured out by now, wrong.
> Others have already mentioned the perf hit you're taking by forcing ASP.NET
> to function in a way it wasn't designed to. But there are ways to get around
> the problem.
>
> First off, I'll assume that you have more content than code. That is, *most*
> of your existing ASP/ASPX pages in the database contain stuff, like invoices
> and whatnot, that do not require intervention by the parser to load and
> display. If this is not the case then stop reading now =)
>
> There are a lot of web applications nowadays that use their databases as
> primary content storage repositories. Blogs are a good example of this.
> However, they do not store code in the database, they store content. If you
> can adapt to this type of scenario then all you have to do is create a sort
> of "master page" or maybe an IHttpHandler implementation that looks at the
> request (even a path), pulls the *content* from the database and then
> displays it on a sort of templated page container. Think of a site like
> MSDN. Lots of articles, yet all the pages look the same. The content is what
> varies. The trick is to simply treat your "pages" as opaque chunks of
> content, and "stream" them into a placeholder page that is rendered the same
> way every time. You can pretty much do anything you want once you're hooked
> into the ASP.NET processing pipeline.
>
> If you do this you'll get rid of your perf problem and you'll still have the
> flexibility and manageability of the database storage. This is web
> application design 101 - always separate content from function and
> structure.
>
> Of course getting from point A to point B could be tricky, but maybe this
> gives you some ideas =)
>
> --
> Klaus H. Probst, MVP
> http://www.simulplex.net/
>
>
> "matvdl" <matvdl@discuss ions.microsoft. com> wrote in message
> news:DF375EB0-8373-4D91-B7F3-0400A8AE22E9@mi crosoft.com...[color=green]
> > DLM,
> >
> > Thanks for your response.
> >
> > I don't believe that SQL is my problem - the time it takes to do the[/color]
> remote[color=green]
> > calls to the external program has not presented any real limitations or
> > caused any significant time-delays. The problem that I have is trying to
> > deal with the quanitity of files that I have and the combiling of those[/color]
> pages[color=green]
> > once they have been returned. Any time issues that I have had are the[/color]
> result[color=green]
> > of parsing and compilling the page.
> >
> > I understand your point - that asp.net was designed to have the pages
> > pre-compiled before requests where made for the pages - I guess this is[/color]
> one[color=green]
> > of the fundamental differances between asp and asp.net. From experiance[/color]
> the[color=green]
> > design that has been implemented had no significant disadvantages when
> > working with asp - in fact I believe that it had some very significant
> > advantages - mainly due to the unlimited number of files it could manage.
> >
> > Question - is it practicale to have 10,000 pages in a single asp.net
> > application. I wouldn't have thought so?
> >
> > I have no alternative but to deal with this many files as it is a legal
> > requirment of the industry I work in is to be able to provide the original
> > copies of invoices to clients - I can't go back to the original data and
> > attempt to re-produce the same invoice just in case the data has changed.
> >
> > So - considering this is what I am stuck with - my problem mainly relates[/color]
> to[color=green]
> > the memory increases - the slowness of the request could be dealt with by[/color]
> the[color=green]
> > more grunt - but the ever increasing memory is more difficult. I believe
> > that this is the result of the creation of a new class each time a new[/color]
> page[color=green]
> > is displayed - although this page gets un-loaded once the info is[/color]
> returned -[color=green]
> > are the increases in memory the result of the new class definition in the
> > asp.net application?
> >
> > If this is the case - is it possible to remove this class definition once[/color]
> it[color=green]
> > has been created? I have tried to delete the file - but this does not[/color]
> appear[color=green]
> > to fix the problem.
> >
> > Does any of this make sense.
> >
> >
> > --
> > matthew
> >
> >
> > "DLM" wrote:
> >[color=darkred]
> > > Go to this link:
> > > http://www.theserverside.net/books/i...ngPerf/pag.tss
> > >
> > > We all know that there is an initial performance 'hit' the first time a[/color][/color]
> ASP[color=green][color=darkred]
> > > .NET Page is requested, and this is true with the page sitting in it's
> > > virtual directory on IIS. What you seem to be doing is increasing the[/color][/color]
> round[color=green][color=darkred]
> > > trip time for any given request/response for a page to include the time[/color][/color]
> it[color=green][color=darkred]
> > > takes to get the page from SQL Server. And this added time is database[/color][/color]
> access[color=green][color=darkred]
> > > time which is probably longer than any HTTP request.
> > >
> > > I have never seen this kind of architecture before. The whole idea is to
> > > serve pages from a highly scalable architecture such as IIS from the[/color][/color]
> very[color=green][color=darkred]
> > > start, and not make the ASP .NET worker process wait for the files[/color][/color]
> before it[color=green][color=darkred]
> > > can dynamically compile pages. In a sense you are using SQL Server as a[/color][/color]
> web[color=green][color=darkred]
> > > server to your web server, which is highly unusual. You need to find a[/color][/color]
> way to[color=green][color=darkred]
> > > place all your pages on the web server and use SQL Server only for data
> > > retrieval/update.
> > >
> > > TAKE A LOOK AT MICROSOFT PATTERNS & PRACTICES
> > >
> > >[/color][/color]
>
>
>[/color]

[matvdl]

RE: Using SQl to store aspx pages and memory problems

DancesWithBambo o,

Thankyou for your help - your response I believe hits the nail on the head.

I am trying to look at my options on pulling the code out of the pages and
placing it in asp.net - not sure exactly how to do it - but are looking at
different options. If you look at one of my recent replies to Klaus I have
outlined a few of my ideas.

I found the asp solution very flexible and it provided a good solution to
the issues I was having - but I was forced to move to asp.net due to
reliability issues and a need to use webservices. Asp pages still get served
occasionally from the server as they were created 3-4 years ago and the
problems I have been having with asp.net have not really caused a great
problem until recently. I guess its time to do some more up-grading.

Thanks for your help.

--
matthew


"DancesWithBamb oo" wrote:
[color=blue]
> To understand why the solution you had working in clasic asp doesn't perform
> in .net, you have to understand the internals of how these systems work.
>
> In asp, every page was intrepreted runtime. It looked for your <% %> tags
> and intrepeted the code between them real time as it ran each page. If your
> page didn't caontain much server-side code and was mostly content then there
> wasn't much to do. Basically the system was designed to look at each page
> each time it was hit as necessary. This is why almost all systems (except
> your) are faster in .net.
>
> .net on the other hand "compiles" every page into IL code when the page
> changes. This IL code is then intrepeted real time by the runtime engine.
> Compiling is a lot of work. It outputs new code, checks syntax, checks type
> casts, checks variable declarations, function calls, and another million
> things. asp would just crap out if it didn't like what it saw. The key here
> is that .aspx files are compiled once for every folder and every folder every
> time a file changes in it. On a normal system, you take a hit once for each
> folder that has a page in it requested. Then nothing changes for a long time
> so the compiler may not do anything for months.
>
> In your system, every time you push a page out from SQL to disk you are
> instructing the compiler that there is a new page that it must compile (not
> to mention the file system write which could possibly be the slowest thing
> you could do on a Windows box). Now if you start loading all 10000 pages
> into the same folder, the work it has to do will get worse as the more pages
> are requested since there will be more pages to compile.
>
> The key to making your system work is to do some major rearchitecting or
> leave it in clasic asp. Why do you want to change? If you rearchitect, you
> need to only store "data" in the database and keep code in files on the
> server compiled. Then you load the pages with data from the db. This is how
> content management systems work.
>
> An example would be your invoice pages. I assume that many of your "pages"
> in the database are invoices. You need to just store them as text along with
> whatever data you need to make the page in SQL Server (table called Invoice?)
> and then have a page called invoice.aspx on the server that uses code to grab
> the data and format it for display(includi ng your graph control).
>
> It really has never been a "good" design to put code in a database. It just
> happens that classic asp was stupid enough that you could get away with it.
> You need to leverage the systems for what they are good at; IIS/.net for page
> serving and SQL server for data storage.[/color]

[Bruce]

RE: Using SQl to store aspx pages and memory problems

Hi,
Convert the aspx pages to controls and use Loadcontrol from a single aspx by
looking up the ID of the SQL stored control! As well you could cache the
controls to improve performance.

regards
Brucek

"matvdl" wrote:
[color=blue]
> I have a system that was originally developed in asp - the pages are saved in
> SQL (there are over 10,000 pages) and saved to a temp directory in the server
> when requested by a client.
>
> I have updated this system and changed the pages that are saved to the
> server as aspx - everything works fine and pages can be served - but
>
> Its not impossible for a single client to request 100 plus pages in one
> session - as each page is requested it is retrieved from SQL saved to the
> temp directory and compiled - problems are.
> - Performance is well down on the original asp system - I believe that
> this due to the compiling of the pages in asp.net
> - The memory usage of server also goes through the roof - there is a
> considerable increase in memory as each page is loaded
>
> Are there any solutions to these problems??
>
> --
> matthew[/color]

[Bruce]

RE: Using SQl to store aspx pages and memory problems

You could also use ParseControl to pass them as a string from the database,
Loadcontrol would load from a file.
Basically in pseudocode:
1. Put a placeholder in a aspx page
2. Get id of stored control from calling page
3. Get Text of ascx in database
4. New Control = Parsecontrol(te xt)
5. Placeholder.Con trols.Add(Contr ol)

Place 2-5 in loop if more than 1 control per page

This way you get one aspx, with 1 or more content controls without redirects
etc.

Regards
Bruce


"Bruce" wrote:
[color=blue]
> Hi,
> Convert the aspx pages to controls and use Loadcontrol from a single aspx by
> looking up the ID of the SQL stored control! As well you could cache the
> controls to improve performance.
>
> regards
> Brucek
>
> "matvdl" wrote:
>[color=green]
> > I have a system that was originally developed in asp - the pages are saved in
> > SQL (there are over 10,000 pages) and saved to a temp directory in the server
> > when requested by a client.
> >
> > I have updated this system and changed the pages that are saved to the
> > server as aspx - everything works fine and pages can be served - but
> >
> > Its not impossible for a single client to request 100 plus pages in one
> > session - as each page is requested it is retrieved from SQL saved to the
> > temp directory and compiled - problems are.
> > - Performance is well down on the original asp system - I believe that
> > this due to the compiling of the pages in asp.net
> > - The memory usage of server also goes through the roof - there is a
> > considerable increase in memory as each page is loaded
> >
> > Are there any solutions to these problems??
> >
> > --
> > matthew[/color][/color]

[Klaus H. Probst]

Re: Using SQl to store aspx pages and memory problems

Matthew,

"matvdl" <matvdl@discuss ions.microsoft. com> wrote in message
news:EFDE3AA2-B54E-4E62-A11F-9A523E4C8067@mi crosoft.com...
[color=blue]
> So - if I have a string of HTML text with a embedded asp.net server[/color]
control[color=blue]
> and I use the response.write function will asp.net recognize that there is[/color]
a[color=blue]
> control with in the document and load it accordingly??[/color]

No, the template parser is long done after you get to write to the HTTP
output stream.
[color=blue]
> If this don't work that only way that I can image doing it would be to
> implement my own method of tags - search from them at run-time and extract
> the code between them and use the Eval function to evaluate the code.[/color]
Does[color=blue]
> not seem like a good solution - but I think it would work.[/color]

You can embed type names or something like that, and then use some sort of
class factory implementation to instantiate the "controls" that need to
render themselves. Pass the HttpContext reference to them or some such so
they can play. Remember that a "control" in ASP.NET is something that just
renders some HTML. They're nothing special and they don't have to be loaded
from disk or have an ASCX extension (although that has its advantages).
Still, avoid that sort of thing as much as you can.
[color=blue]
> Part of my aim is to limit the dependence of the need to store data - once
> the invoice or file is produced ideally I need future requests to only[/color]
look[color=blue]
> at this one file for it to display correctly - therefore the reasoning on
> imbedding the data within the file - but that does not mean that the code
> could be pulled out of the file and placed elsewhere. Having as few
> dependencies as possible on what is required to display the page was part[/color]
of[color=blue]
> the original design and one that ensures that future system changes will[/color]
not[color=blue]
> inadvertently make viewing older invoices fail. Something that has worked
> well over the past 5 years.[/color]

Ideally you never store data with presentation metadata (your markup), but I
will recognize that in some cases it makes sense. But storing the data with
the markup is not a good idea just because you think the application will
change someday - you design applications to be resilient and adaptable to
most foreseeable changes.

--
Klaus H. Probst, MVP

http://www.simulplex.net/