100 thread limit

[pockets]

Ive been wondering this or a while now, but why is the "new posts" link limiting to 100 threads? With as busy as this board is even checking a purple times a day it hits this limit so no telling what I'm missing.

I couldn't find any setting for this in my profile so I assume it is a board level setting.
Posted via Mobile Device

[Liz]

New posts is just that, new posts. Clicking quick links, todays posts may work better for you

[pockets]

Hmm. Every other board I've been on the "new posts" gets new posts since my last visit which is nice if I miss a day or two. Does that link not do that?

[Liz]

Yes it does, but it is based on search. So it runs a search everytime someone uses it. That is why it is limited to 100. Many sites have a posts since you have last visited hack on them. I use it on one that I frequent as well. Here I simply use the todays posts, if I need more than 100. I am a bad judge with that though, as I am tied to the site from wake till sleep in one form or another.

Try the todays posts deal, see if it helps you. If not I would PM Josh and discuss it with him.


edited: I did not realize todays posts was turned back to 100 . At one time it was higher. The results for todays posts and new posts would be the same minus you getting your own post results in there.

[pockets]

Yea, it should search for all new posts since my last visit. I'm not sure what "hack" you are referring to though. The "new posts" seems to work the same across all the other vBulletin setups I frequent.

The today posts obviously won't get the same results, but that's ok. I'll live with it.

[Liz]

The strain on the server to pull up 2k+ posts for everyone who uses it daily, would be insane.. The hack I am referring to is when above your PM notifications, it says blah blah posts since your last visit. Same deal, different place for it to show.

The reason search itself is limited to results as well as the new post search, is the server strain. If the site allows a large number of results it lags and or crashes when several hundred folks are using it. On the smaller sites we have had, we allowed larger numbers. The big ones it just kills server loads

[pockets]

I thought that my be the reason.

Don't take this wrong and I'm not trying to be a jerk, but I would be curious though as to why it would be any strain. When MySql (which I presume is being used) does the query, it has to load everything matching the results anyway. The reason is because it has to sort the results by date and in order to do that it has to find everything that matches, e.g. since the last visit. The fact that the software then limits to 100 results is inconsequential. Because of the sort, it can't just find the first 100 and stop, it has to find everything. Depending on how vBulletin did their code, it probably executes the query twice. One to do the count and one to return the results to be viewed based on the page being viewed (page 1, page 2, etc.). Thankfully MySql is really fast compared to other databases at counting result sets.

Actually, the query for "today's posts" is most likely the same thing as the "new posts", presuming the new posts is based on last visit date.

In any event, I just thought I'd ask

[Josh]

Originally Posted by pockets

I thought that my be the reason.

Don't take this wrong and I'm not trying to be a jerk, but I would be curious though as to why it would be any strain. When MySql (which I presume is being used) does the query, it has to load everything matching the results anyway. The reason is because it has to sort the results by date and in order to do that it has to find everything that matches, e.g. since the last visit. The fact that the software then limits to 100 results is inconsequential. Because of the sort, it can't just find the first 100 and stop, it has to find everything. Depending on how vBulletin did their code, it probably executes the query twice. One to do the count and one to return the results to be viewed based on the page being viewed (page 1, page 2, etc.). Thankfully MySql is really fast compared to other databases at counting result sets.

Actually, the query for "today's posts" is most likely the same thing as the "new posts", presuming the new posts is based on last visit date.

In any event, I just thought I'd ask

You have been told multiple times why we will not raise the limit. It searches a much smaller index and then only pulls 100 results out of a 3+gb table. If it were to pull up more than that it does and will severely impact server resources as it must keep the connection open that much longer. When there are 1500+ users trying to query a extremely large database all at once you can and will have a severe load if you do not limit the number of posts. Besides I seriously doubt you will want to look through all 3-4k posts that would appear if you were not on the site for more than a day. I fully and completely understand how this all works I don't need a primer on how to run a server or website.

Josh

[pockets]

Wow. Ok, I guess my 12 years of software development and experience in building large scale systems has been wrong all this time. 3gb is a small table in my world but that isn't the point.

Most people who run sites don't understand how databases work. Like I said limiting the results to 100 isn't actually doing that. Assuming the software is written correctly it isn't even pulling 100 results, it is only pulling the 20-50 or so for a given page being viewed. Of course that is an assumption on vBulletin's part. Besides, if I was giving a primer I wouldn't have gone into such detail.

Anyway, keep up the good work.

[Torbjorn]

It never ceases to amaze me at the sheer number of people who can always do something better than the people who are ACTUALLY DOING IT!

vBulletin is configurable with many different parameters for accessing the database, this is correct. However, what the masses must realize, the owners of this site have been at it for over 15 years and have PLENTY experience with the habits of users and the configuration of the servers on the backside. While we CAN change the 100 post limit, it has been PROVEN on many occasions to NOT be conducive to the site enjoyment by the majority of our members. If you in fact have the knowledge of programming, databases, etc that you claim to, you already know that you have to make compromises within the program/database to appease the masses (unless of course you're programming for a single user). Thinking any other way is ridiculous.

We welcome your suggestions as always, but when given valid reasons as to why YOUR setup is not going to happen on the site, please give it a rest. Or maybe rewrite the query protocol between vBulletin and SQL.

[pockets]

Originally Posted by Torbjorn

It never ceases to amaze me at the sheer number of people who can always do something better than the people who are ACTUALLY DOING IT!

It never ceases to amaze me how people throw around technical nonsense and stats in an attempt to sound impressive when they probably have no clue what it actually means and people who jump to conclusions.

Did I say I could do it better? No. Did I say you were doing anything wrong? No. Could I write the software better? More than likely yes. However that wasn't the point and nor did I say anything close to the before now.

Could I run this site better? Probably not, nor did I claim to. I actually think the team here does a really good job and I wouldn't have become a paid member (A first for any forum for me I might add) if I didn't find it valuable.

My last response to Liz was exactly "In any event, I just thought I'd ask ". That should have been it, but no Josh had to jump in saying nonsense like 1500 users trying to query the db "all at once". Just because the site says there are 1500 users online they are NOT actually querying the db at once. Unless your definition of "all at once" is say within a couple minute window which is an eternity to a database. A few dozen would be more likely where "all at once" is under a second. Or things like "extremely large" and 3gb. Big yes. Extremely large, don't think so. After explaining it several times, you still seem to think it would actually pull up more than a single page's worth a results at once. If it searches a smaller "index" (I presume you mean table as an index means something entirely different to a db) and then pulls the results out of the main 3gb table then it should be even faster. That would make sense as it could do the search based on date, order those results and then use the primary keys to pull out the actual threads from the main table. But again, it is only going to grab the ones needed to fill a single page. Apparently Josh's definition of "fully and completely understand" is different than mine.

15 years running sites? That's really good, but provides no context. My wife has been running sites for 10 years or so. My dad has been running bulletin boards for over 20 years, before the WWW existed, back when you had to actually dial directly into a board over a 300baud modem and administrating networks for 20 years. Neither one have a clue how databases work nor would I expect them to. Most developers don't even know how a db works. Just as I wouldn't expect anyone here to know how the db works. That is why I mentioned how that particular query works. How that was construed into providing a primer on how to run a site is beyond me.

Originally Posted by Torbjorn

vBulletin is configurable with many different parameters for accessing the database, this is correct. However, what the masses must realize, the owners of this site have been at it for over 15 years and have PLENTY experience with the habits of users and the configuration of the servers on the backside. While we CAN change the 100 post limit, it has been PROVEN on many occasions to NOT be conducive to the site enjoyment by the majority of our members. If you in fact have the knowledge of programming, databases, etc that you claim to, you already know that you have to make compromises within the program/database to appease the masses (unless of course you're programming for a single user). Thinking any other way is ridiculous.

I know all about making compromises in software development. Try building a site that supports 20 million unique visitors a month (compared to the 140k or so for this site) and 200 million page views a month. Yea, compromises I know about. We tuned to find millisecond changes in response times. So "ACTUALLY DOING IT", yea I actually do it. Building web applications is what I do for a living and tuning them is my passion.

If you find it doesn't matter to most people that the list is limited to 100, great, not a problem. Hell I never even asked to have it raised, I merely asked why it was where it was.

Originally Posted by Torbjorn

We welcome your suggestions as always, but when given valid reasons as to why YOUR setup is not going to happen on the site, please give it a rest.

And as I quoted above, I did give it a rest. THREE time. First when I responded to Liz with "...but that's ok. I'll live with it.", then again with "In any event, I just thought I'd ask ", then again when I responded to Josh.

Originally Posted by Torbjorn

Or maybe rewrite the query protocol between vBulletin and SQL.

rewrite the query protocol between vBulletin and SQL? Huh? More technical nonsense... There is no protocol there. It is a string of text in php representing SQL, nothing more. Perhaps you meant mySql instead of SQL? Still even there the protocol between the php engine and MySQL has absolutely nothing to do with the performance of this site or any other one for that matter. Maybe perhaps you mean the SQL that is vBulletin is using to talk to MySQL? Well, SQL isn't a protocol but I'm sure you know that. It's use of SQL probably leaves a lot to be desired but you can't control that without going to the source code nor did I even suggest doing so.

So before you go spouting off a bunch of stats and technical nonsense, save it for the people who have no clue what you are attempting to represent and thus would be impressed.

[Josh]

All I can say is WOW.

You do not know my stats. You do not know my setup nor the way I have things configured. Are pulling stats about the site out of thin air that are nowhere near accurate. So thank you for your "lesson" on how great and knowledgeable you were, however, I don't believe it was needed.

[Torbjorn]

Dayum.....look out boys, I've struck nerve!!

I've been in the real world running networks for Fortune 500 companies for over 25 years and am currently IT Director at my present company. Technical non sense?? I've have had the misfortune of having to deal with 'web developers' for quite some time now and never once (including this time) has it been a pleasant experience. I just love how it seems to me that all you guys know better what we want/need than we do. I have seen web apps that would ONLY run on IE 5.5, seen them when a bug is found the answer was "oh you need opera, firefox, safari or it wont work", and literally thousands of other excuses for piss poor programming. I dont know what kind of web development you do, but in general the quality of that service has gone down over the years.

I did not say that we have been running "sites" for over 15 years. I stated we have been running THIS SITE for over 15 years.

But as per usual, what we have here is a situation that resembles a pissing match. Say what you will about my abilities and knowledge if you want, but I can say that I have been CONSTANTLY employed (read never out of work) for over 25 years in the IT business.

Bottom line here.....JOSH decides whats going to happen. Just as it appears you do on your MANY sites. When someone has a valid issue or suggestion, it is taken under advisement and dealt with accordingly.

What else could you expect from a bunch of old truck fanatics who obviously have no clue about how databases work....

[Torbjorn]

One thing that just doenst sit right with me

rewrite the query protocol between vBulletin and SQL? Huh? More technical nonsense... There is no protocol there.So before you go spouting off a bunch of stats and technical nonsense, save it for the people who have no clue what you are attempting to represent and thus would be impressed.

I wasnt speaking of the standard protocols that web people use (SFTP, IMAP, etc)....rather the "protocols" necessary for communication with SQL. (i.e. http://blogs.msdn.com/b/sql_protocol...owershell.aspx)

So before you go spouting off a bunch of stats and technical nonsense, save it for the people who have no clue what you are attempting to represent and thus would be impressed.

INDEED !!

[pockets]

Originally Posted by Torbjorn

Dayum.....look out boys, I've struck nerve!!

I've been in the real world running networks for Fortune 500 companies for over 25 years and am currently IT Director at my present company. Technical non sense?? I've have had the misfortune of having to deal with 'web developers' for quite some time now and never once (including this time) has it been a pleasant experience. I just love how it seems to me that all you guys know better what we want/need than we do. I have seen web apps that would ONLY run on IE 5.5, seen them when a bug is found the answer was "oh you need opera, firefox, safari or it wont work", and literally thousands of other excuses for piss poor programming. I dont know what kind of web development you do, but in general the quality of that service has gone down over the years.

Well that explains a lot. I've had the misfortune of working for managers that who didn't have a clue how software was built. Didn't have a clue how long something actually takes to build and test and then complain when it fails because they gave unobtainable expectations. You run networks? That is great; my dad did too. I know enough about networks to talk a little on it and ask sort of intelligent questions but I couldn't setup up a corporate level network. Not my area of expertise. Just as, based on your above statement, programming and databases are not yours. As you very well know, the IT world gets far too generalized by the outside. There are network folks, server admins, developers at different levels, etc.

There are a LOT of really bad developers out there, I will admit that. And they make every one else look bad. There are also a lot of really good ones, but they rarely get listened to. I've seen managers and directors spend millions and millions over the years on crappy software and bad decisions. I've seen managers and directors make good decisions. Some I would have gone a different way with, but they were good nonetheless.

Only Microsoft people write code just for IE, and it is a shame and stupid. Or any specific browser for that matter. Developers that make that excuse are piss poor programmers. No doubt about that. Unfortunately developers in the microsoft world tend to be a little short sighted when it comes to good programming practices. Nothing against the MS languages per se, that's just how it seems to happen. php software tends to be the same way for some unknown reason. The language is fine, but companies tend to write really bad in it. Shame. Facebook is the big exception however. They do some good php stuff.

Originally Posted by Torbjorn

I did not say that we have been running "sites" for over 15 years. I stated we have been running THIS SITE for over 15 years.

But as per usual, what we have here is a situation that resembles a pissing match. Say what you will about my abilities and knowledge if you want, but I can say that I have been CONSTANTLY employed (read never out of work) for over 25 years in the IT business.

Bottom line here.....JOSH decides whats going to happen. Just as it appears you do on your MANY sites. When someone has a valid issue or suggestion, it is taken under advisement and dealt with accordingly.

What else could you expect from a bunch of old truck fanatics who obviously have no clue about how databases work....

Apparently none of you actually read anything I wrote. So once again, I'll say it again. I never said any of you were doing anything wrong. Did I? Don't think so. I wouldn't expect you to have knowledge of how the database works. I said originally was that the query doesn't work the way people think it does. It sounds logical that it would limit to X amount of results, but because of the sort, it, meaning the database itself, doesn't really do that under the covers. That was it. Then came all the crap. Apparently my attempt to explain why it shouldn't be slow was read as "you are doing it wrong". Heck I never even said or suggested that the limit should be changed! I just asked why it was there.

Liz said I didn't know anything about vBulletin. I never said I did. I guess you guys can't separate the software from the db as far as the discussion goes. However, I do have access to a vb instance since my dad runs a forum on one of my servers. I completely forgot about that. I'm going to look into it, more so because I hate poorly running software so I'm curious how vb is doing this particular piece of code. I haven't looked at the admin interface in quite some time but if VB suggests using that feature for performance reasons only, then it is a hack on their part.

Originally Posted by Torbjorn

One thing that just doenst sit right with me

I wasnt speaking of the standard protocols that web people use (SFTP, IMAP, etc)....rather the "protocols" necessary for communication with SQL. (i.e. http://blogs.msdn.com/b/sql_protocol...owershell.aspx)

I was pretty sure you didn't mean http, ftp, etc., however you still have it wrong. There is no communication with "SQL". SQL is ONLY a language. That link talks about the network protocols for communicating with SQLServer, which is Microsoft's database. It confuses a lot of people and was probably a poor name choice on MS's behalf. Especially when they write it like "SQL Server" or worse just refer to it as "SQL" instead of "SQLServer". Which makes it look like a server that is serving SQL, which is incorrect. It is a server that interprets SQL and serves result sets. I alluded to that (the php engine to MySQL protocol), but again that has nothing to do with performance of the db or even the site. The protocol itself will pretty much never, ever be the bottleneck. The first bottleneck is almost always the db, usually a result of slow disks and not enough memory. Then hardware, then software. Rarely it will escalate up to the network level. But at that point you are running something like eBay or Amazon or facebook. We should all be so lucky to have that kind of problem.

[Liz]

Originally Posted by pockets

Liz said I didn't know anything about vBulletin. I never said I did. I guess you guys can't separate the software from the db as far as the discussion goes. However, I do have access to a vb instance since my dad runs a forum on one of my servers. I completely forgot about that. I'm going to look into it, more so because I hate poorly running software so I'm curious how vb is doing this particular piece of code. I haven't looked at the admin interface in quite some time but if VB suggests using that feature for performance reasons only, then it is a hack on their part.

I did????

Originally Posted by Liz

Yes it does, but it is based on search. So it runs a search everytime someone uses it. That is why it is limited to 100. Many sites have a posts since you have last visited hack on them. I use it on one that I frequent as well. Here I simply use the todays posts, if I need more than 100. I am a bad judge with that though, as I am tied to the site from wake till sleep in one form or another.

Try the todays posts deal, see if it helps you. If not I would PM Josh and discuss it with him.


edited: I did not realize todays posts was turned back to 100 . At one time it was higher. The results for todays posts and new posts would be the same minus you getting your own post results in there.

Originally Posted by Liz

The strain on the server to pull up 2k+ posts for everyone who uses it daily, would be insane.. The hack I am referring to is when above your PM notifications, it says blah blah posts since your last visit. Same deal, different place for it to show.

The reason search itself is limited to results as well as the new post search, is the server strain. If the site allows a large number of results it lags and or crashes when several hundred folks are using it. On the smaller sites we have had, we allowed larger numbers. The big ones it just kills server loads

I may had replied that and immediately edited it. I have no reason to care to have a pissing match with you. You are doing fine arguing with yourself as it is. Thankfully, I can overlook your superiority complex and focus on the running of the site. If I cared to ignore the kids schoolwork, allow them to care for the baby and blow off my back end work to the staff, I would indeed have time to tear apart your posts and reply. It is pointless though, as you know all and everyone else is brainless. And no, you never said that in words, but you surely have shown us now, huh? Bravo my friend, bravo...

[pockets]

Originally Posted by Liz

I did????

I could be mistaken. I saw a numbered list of statements and that was #1. However I saw it via email on my phone and I don't see it again.

Originally Posted by Liz

I may had replied that and immediately edited it. I have no reason to care to have a pissing match with you. You are doing fine arguing with yourself as it is. Thankfully, I can overlook your superiority complex and focus on the running of the site. If I cared to ignore the kids schoolwork, allow them to care for the baby and blow off my back end work to the staff, I would indeed have time to tear apart your posts and reply. It is pointless though, as you know all and everyone else is brainless. And no, you never said that in words, but you surely have shown us now, huh? Bravo my friend, bravo...

Once again putting words in my mouth. I don't have a superiority complex, but you three sure seem to get upset when someone tries to explain one thing. I know how the db works and that was all I ever stated. Tear apart my posts? With what, wrong information? I can't help if you guys change what I said and think I said things I didn't. I don't know how many times I have to say the same thing over and over.

I never said I knew it all did I? And if you looked at several of my replies I stated that something must be happening on VB's side of things and it is not a db issue, and certainly NEVER EVER said it was something you were doing wrong.

I never said anyone was brainless either. I even said I couldn't run the board any better than you didn't I? I can also guarantee that I don't know squat about networks compared to someone with 25 years experience in them. I'm quite sure that after 15 years you know how the software works from an administrative view very well. I never argued against that.

My third reply should have been the end of the whole thing. I'll save you having to scroll back, here it is in it's entirety:

I thought that my be the reason.

Don't take this wrong and I'm not trying to be a jerk, but I would be curious though as to why it would be any strain. When MySql (which I presume is being used) does the query, it has to load everything matching the results anyway. The reason is because it has to sort the results by date and in order to do that it has to find everything that matches, e.g. since the last visit. The fact that the software then limits to 100 results is inconsequential. Because of the sort, it can't just find the first 100 and stop, it has to find everything. Depending on how vBulletin did their code, it probably executes the query twice. One to do the count and one to return the results to be viewed based on the page being viewed (page 1, page 2, etc.). Thankfully MySql is really fast compared to other databases at counting result sets.

Actually, the query for "today's posts" is most likely the same thing as the "new posts", presuming the new posts is based on last visit date.

In any event, I just thought I'd ask

I bolded the last statement. Then Josh came on and became a complete ass followed by Torbjorn's completely off subject comments. And I'm the one with the superiority complex? wow.

Just to recount, here was my 2nd post:

Yea, it should search for all new posts since my last visit. I'm not sure what "hack" you are referring to though. The "new posts" seems to work the same across all the other vBulletin setups I frequent.

The today posts obviously won't get the same results, but that's ok. I'll live with it.

Bolded that too. I guess neither of those bolded statements qualifies as "letting it go" as Josh told me to do. The only thing I didn't let go were technical errors. I would have no doubt that had I stated something about networks and was completely wrong that Torbjorn would correct me, as I feel he(she?) should. I personally don't like having incorrect information.

The irony is that not one statement made by Josh or Torbjorn refutes my original claim about how the db handles a query with a limit clause that is also sorted. However, just in case I am wrong about that, I'm going to verify with someone whom I know for a fact knows more about mysql and database performance in general than I do.

EDIT: He confirmed my original assertion on the query peformance. Also told me several other things that would affect performance that I didn't know about (learn something new all the time) but not one had to do with limiting or not limiting. So the only conclusion is that VB is doing something that a) it shouldn't be, b) taking a ton of results and for some reason populating another table which would affect performance due to excessive disk usage. Why they would do that is beyond me though.

[Liz]

[Liz]

Ok I have a minute while the kids are at dance and scouts. Against my better judgement, I am going to reply to a few comments.

Originally Posted by pockets

Tear apart my posts? With what, wrong information?

Yes, I have boobs and even some blonde in my hair, therefor I have no brain and will give out all incorrect info. Thank you for guiding me with your vast knowledge.

Originally Posted by pockets

Then Josh came on and became a complete ass

Well usually I call him an ass, but I have earned that right.. and he is my ass. You however have not earned that right. Your comment has earned you an infraction though.

Originally Posted by pockets

My third reply should have been the end of the whole thing. I'll save you having to scroll back, here it is in it's entirety:

You are correct, it should had been YOUR last reply. Unfortunately you refuse to let it go, and are nothing more than an annoying troll at this point.

I stand by my previous replies.. but you surely have shown us now, huh? Bravo my friend, bravo...

[shawno72]

Whoah I feel really dumb, I don't have any clue as to what you all were saying but I do recognize an arguement. I thank this site for all the wealth of knowledge that comes onto my screen when I click the mouse and the keyboard. Without this site there are a ton of things that I would still be in the dark about when it comes to my truck, that is what this site is about, right?? Just glad that the powers that be thought up a cool site for truck guys like me to go to.

[fine69]

Wow...i have no clue how to run a site.

Damn glad this one is here.

And the few things about it that piss me off are not huge enough to cause a ruckus over.

Its mostly just a few jerks here that make me mad anyway.

[powerdriver 1958]

Dear Mr Pockets ,

Just a note from an admittedly Dumb truck driver . I spent 11 hrs putting a 200 ton crane together today . I dragged my tired self in the house , Ate dinner and talked to my wife . Went to the "Cave" , Fired up this ole desk top to read the site and relax for the evening .

Then I see this mess . I think you asked about the 100 thread limit . Which was addressed . If I recall correctly you are the only one that has asked about it . that leads me to believe its not that big of a deal .

For some unexplained reason you start telling The site owners that they are not running the site to suit your ideas . Which is kinda silly . Its their house , They can run it any way they see fit . That should have been the end of it .