87 replies to this topic

[Mind]

I explained what Davidd has found, the 10 second delay.

Response from support ticket

You've got something going on specific to your installation. First guess would be a crashed profile_portal_views table, though I've never heard of it causing a delay like that.

I'd need to take a look at it.. Which means i'd probably need FTP and ACP access.

So it is not something this particular fellow has seen before. Not sure where to go from here. I would like to have our team poke around and see as well.

[davidd]

Dear David, your offer to help is greatly appreciated. We can certainly look at setting up a "sandbox" -- we can put a second installation of the board on the server for testing purposes. Would it be sufficient if we installed a blank copy of the forum software on a subdomain on the webspace and gave you and any other developers ftp access to that installation? Presumably, you would need database access that may be a bit more difficult, but could also be arranged.

Caliban,

It depends on what we are doing. Installing a Mod or creating a Mod to fix a bug or add functionality could be done in blank copy with just FTP access. When it comes to performance tuning, access to the Apache, PHP and MySQL software would be needed to affect changes in that environment. However, those pieces of software can be installed in different directories and run on different ports than the production instances and UNIX accounts could be limited to only those non-production directories.

Some performance tuning or functionality programming might be more difficult with an blank install, but it is better than nothing. We just wouldn't be able to see how the site performance will be if/when changes are made, since the database will be empty and will run fast no matter what we code, even if it is inefficient. We can generate dummy data to help with some of this, which is again better than nothing, but not as good as the real deal.

The only issue is that you'd have to work with a blank slate - we cannot "run some SQL to remove sensitive data".

Can I ask why this is not open for discussion? Or are you saying you aren't sure how it would be done, technically?

If solution does not work, we'd need proper contracts with those who will have access to the data. Again, as the board is keen to resolve technical issues, we could spend some of ImmInst sparse treasure to make sure that people who help out on this are not left out of pocket.

Contracts would be fine. I posted about some possible ideas on how to pay people and how to prioritize change requests in another thread. My main motivation for the fixing performance issues is not money. I get the feeling the same is true of other people who have volunteered.

...
It seems like we have a number of people who are willing to volunteer, so I don't think money is the issue. However, if it does come down to money, how about people donating money for various functionality and giving the programmer that money? That may be the best way to prioritize the importance of the requests. If nobody is willing to pay for it, then it may not be all that important?

In addition to the pay per function model, members (have to be a member, not just a registered user) could be given a pool of imaginary money to spend each month. They could decide which enhancements to spend that imaginary money on and that could be used to prioritize the items that didn't have *real* money donated (if real money donations idea is implemented). More priority should be given to real money than fake money.

This type of limited weighting of items is generally a better mechanism than a model that relies on assigning a value of 1 through 5 or 1 through 10 to an item. In those models, you end up with a lot of ties and people never really know what a number in the middle means. A limited voting model serves to show just how important something is compared to other items.

Obviously, there would need to be a number of discussions and expectations laid out, but you get the basic idea.
...

(Personally, I'd probably have a more radical preference: if someone could suggest a way of migrating away from IPboard without losing any data or functionality, this may help with better CMS and subscription integration.)

Interesting. What board software are you thinking of? phpBB? That's the one I'm thinking of setting up myself, but there are a number out there. Is there one that you know of that integrates better with various CMS software? Are you thinking of staying with Drupal or going with some other CMS?

The sandbox environment allows for testing out all sorts of things like this. Of course, such things can be tested on our own, personal machines too, but if you want to test converting the data, that's best done on the same machine.

There are conversion tools that people have written. I've seen some IPB to phpBB converters, for instance.

What is it that you would like to see in another board that isn't available or is difficult with IPB? Is it strictly integration with the CMS? What type of integration are you thinking of specifically, other than subscription? And what is it about the subscription that isn't well integrated? I'm asking, because maybe that could be something we could fix if it is not working well right now. I'm also trying to learn from real work examples, since I'm thinking about setting up a system myself.

David

[davidd]

I am all for getting a team of people together: Maruisz, Davidd, Maestro949, maybe Harvey. Those are who I have in mind. I would like Lightowl to organize and lead the effort, however, he seems to have disappeared. Anyone hear from him lately? I am sure you will appreciate having some sort-of direction and having a person responsible for reporting changes, progress, such things. If I don't hear from Lightowl by Thursday I will push to give ftp, sql, shell access to someone else or the whole group.

Yes, direction and coordination are important. I'd recommend maybe a couple "admins" (who could also be programmers), who could give various access to people (in the sandbox environment). With at least a couple people, if someone goes on vacation, we won't be left hanging. People do go on vacation -- nothing wrong with that.

We'd have to see how the coordination effort would work best. Having one person for people to report to does provide for single point of organization, but that can also be a bottleneck and could suffer when that person is unavailable. Tough call.

It is too bad we don't have unlimited test licenses for IPB (or if we were using free board software). That way we could install locally on our own machines, test out changes, and then promote to the sandbox. Having a bunch of people making changes in the sandbox at the same time could prove problematic. Sandboxes are only so big and you want to keep the sand on the inside.

I think we might want to set up an automated process that backs-up the IPB source code to other directories every night. That way, if someone puts some code in place that completely breaks it, we can revert back easily and we don't have to hunt to find what was changed. We can use "diff" (unix tool that compares text files) or some other similar tool to see what was changed by comparing to the backup(s) too.

I'm sure we'll find what works as time goes on. The main thing is to try.

David

[davidd]

I explained what Davidd has found, the 10 second delay.

Response from support ticket

You've got something going on specific to your installation. First guess would be a crashed profile_portal_views table, though I've never heard of it causing a delay like that.

I'd need to take a look at it.. Which means i'd probably need FTP and ACP access.

So it is not something this particular fellow has seen before. Not sure where to go from here. I would like to have our team poke around and see as well.

There is a command to repair crashed MySQL tables and other commands to fix all tables or check all tables for corruption. That's probably what he'd be doing. Is there a problem with him having access?

I can step someone through some commands to run to check the tables if it is preferrable to have our own people looking at it. But I'm guessing the IPB people are a bit more efficient in knowing which tables/config to look at, since this is what they do every day.

My original idea was to look at the table types (which engine they use) and see if they were set to MyISAM (which is notorious for locking issues) or InnoDB. There is also a HEAP option that puts tables in memory. IPB does recommend that the ifb_sessions table be set as HEAP, but I'm not sure how that would help with locks, because HEAP tables do still lock, just for a shorter period of time if being table scanned, since they are in memory, which is faster than disk access.

There are commands to check which tables are locking too.

The crashed table is an interesting theory. If there is a piece of code that locks one table and then tries to access another table and that second table is crashed, then it might take a while to timeout or it might retry for a certain period of time before giving up. After giving up/timing out, it might go back and unlock the original table, thus letting other queries/updates continue on that table and those queries/updates could be from other forum pages.

If we get a little more organized on the development front, we can put together some automated jobs that can check for problems nightly. It may just be that we've accumulated a number of issues over time that haven't been noticed yet in log files, etc.

If it is table locking, the best bet is a table that all the pages query/update. Changing such a table to the InnoDB engine would relieve the locking issue, but not the root issue of why the table is being locked for an extended period of time. Again, this is all in the context of *if* it is table locking.

Oh, one other thing to ask him is "What is going on in the user profile when each of the tabs is clicked and the ajaxload image is displayed? What tables are being accessed and is there some sort of retry/polling mechanism in there?" (ajaxload is referring to the "Loading Content..." popup that is displayed when the user profile tabs are clicked on)

Thanks for taking the time to submit the original ticket with them. That's progress!

David

[Mariusz]

I think we might want to set up an automated process that backs-up the IPB source code to other directories every night. That way, if someone puts some code in place that completely breaks it, we can revert back easily and we don't have to hunt to find what was changed. We can use "diff" (unix tool that compares text files) or some other similar tool to see what was changed by comparing to the backup(s) too.

We should set up version control system (i.e SVN) and use that to control any changes being made to the system. 


Or better yet something similar to copy of the system I'm using at work. 

1. Set up repository that will keep track of any changes made to the website

2. Set up domain dev.imminst.org (development, password protected)

Here we will have a working copy (current version of whatever is stored in the repository. This will be accessible to developers, and allow them to make any changes they need. Database used will be an empty set with some dummy date for testing purposes. If/when the site is ready to be deployed we checkout current stable version of the website to a new directory.

3. set up domain stg.imminst.org (staging)

This could be accessible to anyone. Here we will have current version of the page available for testing puroposes. It would use copy of the real data with sensitive information removed for security purposes (if neccessary). Anyone could use this page to test new changes and impovements. If page is working here without any problems we move to the next step.

4. imminst.org

Production website. No changes can or should be made on this copy of the website, and no change should be made to the database it uses (unless they have been tested in dev and stg first)


Pluses of using this setup and SVN version control:


1. change can be reverted if they are causing problems.

2. more then one person can work on the website, even on the same file. Changes are then merged by the administrator.

3. security - we can check any change made to the system by any user allowed to make changes. This way it is much harder to hide malicous piece of code in the software. 

4. this is additional form of backup. we can always revert changes made even few years back, if they are proven to be security problem of are simply not working.

5. not one person has to work on the main database. database is not changed unless it is proven that the changes will not cause any problems.

6. if someone has to quit for any reason, or new person wants to join this effort it's as easy as giving them access to the svn where they can checkout newest version of the software, change it and import changes to dev server. Only admin will have access to the current dataset with all sensitive informations. There is no need to give anyone ftp access, shell etc. just one person - administrator.. Whos resposibility will be to manage repositories, merge changes and make sure that the server is working correctly. 



Mariusz

[caliban]

I think the themes and suggestions emerging here are very promising, but I for one am loosing track of what we can do to help you guys get on with it.
I set up a subdomain called "webhelp" yesterday and started looking at copying data, but I realised it is much better if one of the developers would do that.

So:
If there is a team in place (rather than a single individual) and members of the team are willing to sign a legally binding agreement not to abuse their position - then as far as I am concerned that team can have all the acess it needs.

I would propose to ask lightowl (who has already been given all revelant details), to set up whatever is required to assist the developers.

This would need to be authorised by the whole board which takes at least 48h hours.

If that sounds useful, could people willing to help confirm their intention here please?

[brokenportal]

With a legally binding agreement then I will vote yes on authorization to the people that anybody on the board decides on.

There is a list of people that are interested in helping with this kind of stuff here: are you interested in helping with imminst forums programning?, indicate your interest in that, & to more programmers on board here

[davidd]

I think we might want to set up an automated process that backs-up the IPB source code to other directories every night. That way, if someone puts some code in place that completely breaks it, we can revert back easily and we don't have to hunt to find what was changed. We can use "diff" (unix tool that compares text files) or some other similar tool to see what was changed by comparing to the backup(s) too.

We should set up version control system (i.e SVN) and use that to control any changes being made to the system.

Yes, that's certainly the best plan in the long term. In the short term, we might have to get by with a nightly copy of the code. And even if we use a source code control system, it would be nice to have a nightly or weekly copy to revert back to quickly in case an addition messes up the sandbox instance.

3. set up domain stg.imminst.org (staging)

This could be accessible to anyone. Here we will have current version of the page available for testing puroposes. It would use copy of the real data with sensitive information removed for security purposes (if neccessary). Anyone could use this page to test new changes and impovements. If page is working here without any problems we move to the next step.

We can't do this currently. We only have 1 test license (unless we have a special deal with IPS). However, I'd suggest it is worthwhile for a director to talk to IPS about getting more test licenses. Just explain that we are organizing geographically disparate developers to help out with the site and that we'd like them to have the ability to do unit testing on their own machines before putting the code on the main sandbox.

2. more then one person can work on the website, even on the same file. Changes are then merged by the administrator.

This assumes we have local instances. We can't really work on changes without testing them. In other words, they need to be tested before being checked in.

3. security - we can check any change made to the system by any user allowed to make changes. This way it is much harder to hide malicous piece of code in the software.

Agreed. Code reviews would be nice.

6. if someone has to quit for any reason, or new person wants to join this effort it's as easy as giving them access to the svn where they can checkout newest version of the software, change it and import changes to dev server. Only admin will have access to the current dataset with all sensitive informations. There is no need to give anyone ftp access, shell etc. just one person - administrator.. Whos resposibility will be to manage repositories, merge changes and make sure that the server is working correctly.

For the highlighted part, I assume you mean for the production instance and not the sandbox?

I would recommend at least two people for any roles that are created, so all our eggs aren't in one basket.

Very good recommendations!

David

[davidd]

I think the themes and suggestions emerging here are very promising, but I for one am loosing track of what we can do to help you guys get on with it.
I set up a subdomain called "webhelp" yesterday and started looking at copying data, but I realised it is much better if one of the developers would do that.

Thanks for getting the ball rolling, caliban!

What data are you talking about? The MySQL database data, or do you mean source code files, etc?

I would propose to ask lightowl (who has already been given all revelant details), to set up whatever is required to assist the developers.

Agreed...as long as he is around. I thought he went missing?

We need to figure out from the hosting company if there is any problem with us installing another instance of Apache, MySQL and PHP. They may have a certain way they want these set up. They may require that they do the setup. We also need MySQL to run on a different port, point to different directories, etc.

It is possible with some of these products to use the same binaries and just have different config files or different lines in the same config files, but I would suggest it would be better/safer to separate out the sandbox from the production instance as much as possible. That way we can shut them down, bring them up, change config, etc., etc. without affecting the production/live instance.

If that sounds useful, could people willing to help confirm their intention here please?

I am willing to help. At the moment, I'd rather help with performance tuning. I'd be up for functional debugging too. In the future I may be up for functional enhancements.


In addition to the fine recommendations made by Mariusz, I think we still need to come to a decision on the method of making changes. My vote is for creating our own Mod packages and using the IPB Universal Mod Installer framework. As mentioned on that page, the advantages are:

...

What is it?:
The Universal Mod Installer is a script designed to let you quickly and easily install or uninstall any mods that are designed to be compatible with it. Using this script, you can have one secure place to perform (nearly) all of the steps involved in adding or removing modifications to your v2.2.x or v2.3.x Invision Power Board.

What can it do?:
The script reads the XML file that will come with the mods you want to install and use this information to build a task list of steps that it must perform. This task list will be dynamically generated based on the content of the XML, and it will use that same info to construct a list of uninstallation tasks if you ever want to uninstall a mod using the script. The following is a list of what things the script is capable of doing/creating:

• ACP Setting Groups
• ACP Settings
• Components
• Language Strings
• Skin Templates
• Tasks
• Help Files
• ACP Help Entries
• Database Tables
• Database Columns
• Database Rows
• Database Updates
• Custom Scripts (mod makers can write PHP files with custom mod installation instructions not covered above)
• Rebuild IPB Caches (Rebuilds the Components, Forums, Groups & Settings caches)
• Rebuild Skin Caches

What can't it do?:
The script is very powerful, but it won't do all of the work for all mod installations. It cannot perform file edits (with the exception of the Language and Skin files), those will still need to be done manually. The only other limitation I am currently aware of is that the code to allow Database Tables to be created is supported in MySQL only, not in MSSQL or Oracle. MySQL is far and away the most commonly used database driver, though, so this should not be a problem for most users. And of course it is only going to work with mods designed to use this script; mod makers would need to create their XMLs in a very specific structure to take advantage of this.


Benefits:
So what exactly are some of the benefits to using this script?

• Security - This script is now fully integrated into the IPB Admin CP, meaning that someone would have to find a way to log in to your Admin CP before they could access this script; it adds no additional security risk to your board.
• Convenience - There's no need to remember or bookmark any URL to an installer script anymore, as long as you can reach your Admin CP you can install, uninstall, or upgrade any compatible mods. No more worrying about whether or not you typed in the correct URL to the installer, and no more asking the mod maker for uninstallation instructions.
• Thoroughness - The feature set of this mod is second to none, it can perform a wide variety of different tasks, and the ability to add custom PHP scripts into it just increases the flexibility of potential uses of this script.
• Staying up to date - The Admin CP menu for this component will keep you updated of new versions for the mods you have installed using this script, as well as keeping you updated on the script itself.

...

What version of IPB do we have?

Thanks,
David

[Mariusz]

Yes, that's certainly the best plan in the long term. In the short term, we might have to get by with a nightly copy of the code. And even if we use a source code control system, it would be nice to have a nightly or weekly copy to revert back to quickly in case an addition messes up the sandbox instance.

If we will be using version control, then we need only daily  (hourly) backup of the database. Everything else could be checked out from repository. 
And from my experience term "best plan in the long term" means "hell no". I think if we are to make this work we have to do it correctly. This is also to protect us from any problems caused by other developers. If I'm required to sign a legal agreement, then I would like to know that no one can blame any malicious changes on me. 

We can't do this currently. We only have 1 test license (unless we have a special deal with IPS). However, I'd suggest it is worthwhile for a director to talk to IPS about getting more test licenses. Just explain that we are organizing geographically disparate developers to help out with the site and that we'd like them to have the ability to do unit testing on their own machines before putting the code on the main sandbox.

Well, then we will have to set this thing up  differently. Plus don't forget - forum is only part of the website, the rest could be set up as I'm proposing. 

6. if someone has to quit for any reason, or new person wants to join this effort it's as easy as giving them access to the svn where they can checkout newest version of the software, change it and import changes to dev server. Only admin will have access to the current dataset with all sensitive informations. There is no need to give anyone ftp access, shell etc. just one person - administrator.. Whos resposibility will be to manage repositories, merge changes and make sure that the server is working correctly.

For the highlighted part, I assume you mean for the production instance and not the sandbox?

No, I think that we should have only one person with administrative rights. 

I would recommend at least two people for any roles that are created, so all our eggs aren't in one basket.

I don't agree with that. I can work with anyone, as long as that person will document everything, and be responsive to suggestions and improvement proposed by other team members. I think it's a dangerous idea to allow more then one person at the time to have full access to the server and I don't even want to touch any server that has more then one admin.  If there is more then one preson responsible, then no one is responsible. We will need a gatekeeper, that will keep track of ALL changes made to the server, even if it's a small thing like uploading new favicon. 


Mariusz

Edited by Mariusz, 04 March 2009 - 10:40 PM.

[davidd]

Yes, that's certainly the best plan in the long term. In the short term, we might have to get by with a nightly copy of the code. And even if we use a source code control system, it would be nice to have a nightly or weekly copy to revert back to quickly in case an addition messes up the sandbox instance.

If we will be using version control, then we need only daily (hourly) backup of the database. Everything else could be checked out from repository.
And from my experience term "best plan in the long term" means "hell no". I think if we are to make this work we have to do it correctly. This is also to protect us from any problems caused by other developers. If I'm required to sign a legal agreement, then I would like to know that no one can blame any malicious changes on me.

Mariusz. I didn't mean "not going to happen". I just meant we could do it manually until we had time to install the source code control system. We could manually send the Mod files to one of the admins to install in the sandbox. Again, this assumes we can test them out at home first, which is only possible if we have IPB at home.

I'm not suggesting we do this without some form of source code control. Just that it may be manual until we get the source code control software in place. If someone wants to take up that challenge of picking one out and getting it set up, all the more power to them!

We can't do this currently. We only have 1 test license (unless we have a special deal with IPS). However, I'd suggest it is worthwhile for a director to talk to IPS about getting more test licenses. Just explain that we are organizing geographically disparate developers to help out with the site and that we'd like them to have the ability to do unit testing on their own machines before putting the code on the main sandbox.

Well, then we will have to set this thing up differently. Plus don't forget - forum is only part of the website, the rest could be set up as I'm proposing.

I'm hoping we can get more tests licenses. And yes, that is a very good point. Apache and MySQL and Drupal can be installed at home free of charge. We'd just need access to the website files to create our own test bed at home. Anything that is in the board or is for integration between the non-board and the board would require access to IPB software.

6. if someone has to quit for any reason, or new person wants to join this effort it's as easy as giving them access to the svn where they can checkout newest version of the software, change it and import changes to dev server. Only admin will have access to the current dataset with all sensitive informations. There is no need to give anyone ftp access, shell etc. just one person - administrator.. Whos resposibility will be to manage repositories, merge changes and make sure that the server is working correctly.

For the highlighted part, I assume you mean for the production instance and not the sandbox?

No, I think that we should have only one person with administrative rights.

Again, this works if we each have an instance at home. Otherwise, I don't see how it could work.

I would recommend at least two people for any roles that are created, so all our eggs aren't in one basket.

I don't agree with that. I can work with anyone, as long as that person will document everything, and be responsive to suggestions and improvement proposed by other team members. I think it's a dangerous idea to allow more then one person at the time to have full access to the server and I don't even want to touch any server that has more then one admin. If there is more then one preson responsible, then no one is responsible. We will need a gatekeeper, that will keep track of ALL changes made to the server, even if it's a small thing like uploading new favicon.

What happens if that one person gets hit by a bus or goes on vacation for a month or just doesn't feel interested in doing the work anymore and doesn't notify anyone? All the tracking documents could be shared by two people. It isn't like they are going to keep track in their heads.

In the real world, there aren't many systems with a single admin. Look what happened in Jurassic Park with that model.


Mind/caliban/brokenportal - Has anyone been able to reach out to IPS to see about getting more tests licenses?


David

[Mind]

From the IPB knowledge base:

Am I Allowed to Have a Test Board?

We strongly encourage our customers to have a test board. This is an excellent way to test out the features of the board and test skins and modifications without worrying about affecting the real board. You may have one test board, however you must make sure that the board is not accessible by the public and the board is turned off.

Not sure how to set it up. Will have to be done by the engineer team.

[davidd]

From the IPB knowledge base:

Am I Allowed to Have a Test Board?

We strongly encourage our customers to have a test board. This is an excellent way to test out the features of the board and test skins and modifications without worrying about affecting the real board. You may have one test board, however you must make sure that the board is not accessible by the public and the board is turned off.

Not sure how to set it up. Will have to be done by the engineer team.

Mind,

Yes, that's the one license we have for a sandbox. What we want to know is if we can negotiate to have serveral test boards for developers to unit test against. These would not be available to the public. I'm not sure what they mean by "the board is turned off". It has to be on to test with it, but it doesn't have to be accessible to non-developers. I'm guessing IPB has a feature to not allow accessing the board unless you are logged in and then we can limit the logins to only the developers. I'd have to look in the admin control panel documentation to find the switch, but it has to be there.

This isn't a technical issue. We have the software and could technically install it on our own machines. This is a legal matter, since we have a specific license that only grants us one test license. I'd say, ask for 10 and let them know that we have no plan to use the 10 for live systems. We just need it for disparate, volunteer developers to help fix bugs, create functional enhancements and performance tune the system.

David

[Mind]

Ok, I asked about multiple test-beds.

No answer on the ajax pop-up loading thingy. They just repeated a call for access. I don't have too much problem with having them poke around. I just thought it would be nice if you guys could kind-of follow along. Caliban has asked for input from the other directors. No word from Lightowl yet. The slow wheels of Imminst are moving. Soon we will get this thing cracked open and figured out.

[davidd]

Ok, I asked about multiple test-beds.

No answer on the ajax pop-up loading thingy. They just repeated a call for access. I don't have too much problem with having them poke around. I just thought it would be nice if you guys could kind-of follow along. Caliban has asked for input from the other directors. No word from Lightowl yet. The slow wheels of Imminst are moving. Soon we will get this thing cracked open and figured out.

Great! Thanks for getting the ball rolling. We'll just have to wait to see what they say. It is too late now, but if I would have been thinking, I would have suggested to phrase it as, "We are comparing our options for our website and whether to keep using IPB. One of the differentiators between IPB and other products is how many test beds we could have. With IPB, we only have one. With other products, we can have an unlimited number. We are wondering if we can have more test beds with IPB than the default licensing allows for." Not sure if there will be a chance to do that, unless they ask about our intentions instead of giving us a yes/no answer.

I'm willing to "follow along" with them if given access or whatever you had in mind. Just let me know.

David

[Mind]

Well, if the IPB guy finds something, he is going to respond in some tech terms that I don't fully understand, and probably not leave a good trail of what was modified.

If he doesn't find something right away, then he might spend a couple messin' around and who knows all the things that might be changed - on our live forum.

Therefore, I would like to wait a couple days, give you(davidd), mariusz, maestro949, lightowl, etc... access, and then you can work it out safely and and/or help IPB support find the issue, or at the very least, translate what IPB tech help does into English.

[davidd]

Well, if the IPB guy finds something, he is going to respond in some tech terms that I don't fully understand, and probably not leave a good trail of what was modified.

If he doesn't find something right away, then he might spend a couple messin' around and who knows all the things that might be changed - on our live forum.

Therefore, I would like to wait a couple days, give you(davidd), mariusz, maestro949, lightowl, etc... access, and then you can work it out safely and and/or help IPB support find the issue, or at the very least, translate what IPB tech help does into English.

Okay, I think I understand. Like I said, I'm willing to do whatever I can.

An alternative approach, in this case (because I already have at least an idea of what is going on), is that I could give you some commands to run and you could report back the results. These aren't commands that would change anything. They just provide information. They are some of the same commands that he would run, most likely. Let me know if you would like to do this and I can put together the commands. Depending on the results, then there may be other commands to run to actually affect change.

If not, then I guess the next step would be to:

A) Go through whatever process is necessary to give us access
B) Give us a day or two to look into it
C) If needed, let the IPB guy in and ask him to document everything and anything he does that *changes* anything (commands that just return information like I suggested above don't fall into this category).
D) Report back, in english, what happened and what was done to fix it, along with the technical details as well.

Please let me know which way you'd like to proceed from the above, or if you had something else in mind.

Thanks again for working on this issue!

David

Edited by davidd, 07 March 2009 - 06:55 PM.

[Mind]

The access question is going through a vote as we speak.

IPB rep said we can create multiple test beds as long none of them are "live" or used to support a separate community.

Hello,

Yes, that is fine.

Regards,

Debbie Mecham
IPS Customer Relations
Peterborough, UK
Invision Power Services, Inc.

[davidd]

The access question is going through a vote as we speak.

IPB rep said we can create multiple test beds as long none of them are "live" or used to support a separate community.

Hello,

Yes, that is fine.

Regards,

Debbie Mecham
IPS Customer Relations
Peterborough, UK
Invision Power Services, Inc.

That IPS reply is truly great news. Our hands would be significantly tied without this option. That will help out with functional enhancements the most, since people can develop the stuff on their own machine, test it, and then it can be applied to the "true" sandbox for integration testing (making sure it works with everything else in the sandbox at the time). It will reduce how many people need shell access to the physical machines that also serve up the live website.

The particular performance issue we have right now in the "live" board (10 second delay) requires probing the live system. It is not likely that we'll be able to reproduce the issue in our test beds. That doesn't meant that all performance tuning in the future will require looking at the live system, but this particular one is an oddball. It is one of those "shouldn't happen" issues. However, the good news is that some very simple commands can be run to check on the health of the database tables, which will tell us if we are sniffing in the right direction.

We'll want someone to put together a good install doc so that all the test beds can be installed the same way (which should also be the same way the live site is installed), where to get the code/config files, etc., etc.

The 10 second issue is more of a time sensitive thing. However, for true development efforts on other tasks, I strongly suggest coming to some decisions/conclusions on our development methodology *before* people dig in.

Good job Mind!

David

Edited by davidd, 10 March 2009 - 10:07 PM.

[Mind]

Methodology/priorities: top priority is the slow-loading, but there are other bugs that have developed and these should be taken care as well, before any other modding goes on.

The Board has authorized the creation of a team of engineers. I will nominate a lead engineer today or tomorrow.

The people who responded in this thread will be invited to join the team and I suppose the first thing we need to do is dole out access. Someone mentioned the need for IPB admin access and FTP access. Also I think Maruisz mentioned shell access. Could someone let me know precisely what that entails. Server CP? MySQL access?

[Mariusz]

Also I think Maruisz mentioned shell access. Could someone let me know precisely what that entails. Server CP? MySQL access?

Shell access means access to the command line. This will be needed to edit config files, restart services etc. 


Also what is the current situation as far as backups of the system? Is anyone monitoring that?



Mariusz

[Mind]

I have asked our ISP about back-ups but no response as of yet. I am sending another email right now.

[Mind]

Newest problem: errors in the Imminst Drupal installation. None of the Imminst Drupal pages are working. I have kamil at helldesign looking into it, but unsure when or if he can come up with a solution.

I am not sure who around here has in-depth knowledge of Drupal (maybe it is a database problem), but this is now the top priority. Imminst has some important info in the Drupal pages and it is the public face or "front door" to the website. It might be a simple fix.

2nd would be the slow loading forums and general performance issues, IPB and MySQL.

3rd would be bulk email problems.

I have a list of other things lying around on my desk somewhere.

I have nominated Maestro949 as lead engineer. I would like to bring Lightowl, Maruisz, and davidd onto the engineer team as well. There may be others in the near future. Hopefully we can get this rolling next week.

[Mind]

Oh yeah

Put this on the list: Uploading attachments/pictures function is broke. Happened within the last couple of days.

The Drupal problems started on Wendesday or Thursday (March 11th/12th) time frame.

[Mariusz]

I have nominated Maestro949 as lead engineer. I would like to bring Lightowl, Maruisz, and davidd onto the engineer team as well. There may be others in the near future. Hopefully we can get this rolling next week.

Hail Maestro949! We who are about to start coding salute thee!
Awaiting my orders...

Mariusz

[davidd]

The people who responded in this thread will be invited to join the team and I suppose the first thing we need to do is dole out access. Someone mentioned the need for IPB admin access and FTP access. Also I think Maruisz mentioned shell access. Could someone let me know precisely what that entails. Server CP? MySQL access?

Here's one way to do this. We can break it down into the various systems we plan on setting up for development.

1) We need some number of people to have access to the Production (live) site to investigate oddities that only happen there and for general maintenance of that system. These people would also update the site with the new code releases or make any other changes, that have been tested elsewhere, into the live site.

2) We need some number of people to have access to the pre-Production site -- let's call this one "Staging". This is the system where we "stage" the code, to make sure all the various changes work together properly, before moving it into the Production site. The people who have access to this site would incorporate new code submitted by developers, install 3rd party Mods (if we decide to use them), install other software that we want to incorporate into the site, etc., etc..

3) We need some number of people to install development environments outside the hosting servers. This would most likely be on their own machines. These people would make Mods, test out other software, etc., etc.. After they feel it is working properly, it would then be submitted to the people managing Staging for integration testing with the other pieces of code.

Now, I personally consider the above to be the minimum breakdown for smooth development. An improvement on this, if we are willing, would be to have a "Test" environment that sits along-side Staging and Production. This environment would be where the most recent code changes can be merged together for testing. Staging would then be just for code that has been tested and verified to work together, for one last pass of testing, before going onto Production.

The real benefit of a Test environment is that it allows continued development after putting the code into Production. If a problem is found, and a small number of changes are required, those can be made and moved into Staging for testing. Then Staging can be re-deployed into Production. Without a Test environment, Staging may contain a bunch of changes already that aren't fully tested, so we wouldn't want to co-mingle the bug fix with the new development and have to push it all into Production.

In corporate America (and elsewhere), you'll generally find more environments for development/testing in more established IT groups, because over time they find that there is increased flexibility with more environments.

Oh, and as has been mentioned before, we'll want to make some decisions on how exactly we package our code changes (my preference is Mods, using the Universal Mod tool for IPB) and how we do source code control.


We need to start making decisions about things like this before we can decide what kind of access we need. We can also break down our goals a bit -- that may help too.

A) We have current production issues with performance and maybe some other critical bugs.

B) We have longer term development work (functional enhancements and defect fixes) and longer term performance tuning.

Once we decide on our environment layout (Devs, Test, Staging, Production), then we could have the people who have access to Production work on (A), technically, before we have the Devs/Test/Staging all set up. I don't think we want to work on (B) until we have the Devs/Test/Staging all set up.

If people are okay with the above, then we can talk about what "having access" means for Test/Staging/Production. And someone can decide who those people are.

Comments?

David

[Mariusz]

Oh yeah

Put this on the list: Uploading attachments/pictures function is broke. Happened within the last couple of days.

The Drupal problems started on Wendesday or Thursday (March 11th/12th) time frame.

It couldnt just happen for no reason. Do you know who was playing with the server and why?


Mariusz

[Mariusz]

Newest problem: errors in the Imminst Drupal installation. None of the Imminst Drupal pages are working. I have kamil at helldesign looking into it, but unsure when or if he can come up with a solution.

Well, it looks like someone with access to the server did something they shouldnt have done. No the question is who, why, and what. Also did you get an answer from isp regarding backups? Most likely they don't have it, and whoever was playing with the server did not create backup of files modified. 


Mariusz