There is actually a very simple workaround for this if you use Google App Engine for Java. Suppose you have the following static file:

/static/big_javascript_lib.js

In your .html files, you reference that the static file like this:

<script type=”text/javascript” src=”/static/big_javascript_lib.js“></script>

And everytime the browser will load a html page containing the static file, Google App Engine will return status code 200 with the full content of the file, even if the content was not modified since the last time the browser requested it.

The workaround is very simple. First, in “WEB-INF/web.xml”, define a servlet for path “/304_enabled/*” (you can use any path for which no static content exist):

<servlet>
<servlet-name>StaticServlet</servlet-name>
<servlet-class>com.opcode.servlet.StaticServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>StaticServlet</servlet-name>
<url-pattern>/304_enabled/*</url-pattern>
</servlet-mapping>

Then, create the StaticServlet class like this:

package com.opcode.servlet;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class StaticServlet extends HttpServlet {

// This static file handler will correctly return a 304 (Not Modified)
// response code when appropriate. This is not the case with the default app
// engine static file handler.

public static final String PATH = “/304_enabled”;

@Override
protected void service(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
String file = request.getRequestURI().substring(PATH.length());
request.getRequestDispatcher(file).forward(request, response);
}
}

Finally, in your .html files, prepend “/304_enabled” to the references of all all static files like this:

<script type=”text/javascript” src=”/304_enabled/static/big_javascript_lib.js”></script>

You must NOT move your static files in a “/304_enabled” folder, let the static files where they are, the servlet will find them.

Now the servlet will correctly return response code 304 if the static content was not modified since the last time the browser requested it.

UPDATE: This issue is now fixed, so the workaround is not necessary anymore. Thanks Google.

In addition to those 6 games, we also added a game list page that shows all games implemented to date on Good Camel Games. From that page, you can easily start any game. That page will be updated as games are added.

What is good about Good Camel Games is games are fun, simple and very intuitive. No need to invest a huge amount of time just to understand the game. The fun starts in seconds.

UI Improvements – Now the UI is more consistent and easier to use. A Good Camel logo has been added and each page is properly identified with a title. There is still room for improvements in that area. Suggestions are welcomed.

Facebook Integration – Good Camel Games has a better Facebook integration. Now you will see who of your Facebook friends are also online on Good Camel Games. With the “Invite Friends” feature, you will also be able to invite any Facebook friend to play in a matter of seconds. And you do not need to go to Good Camel Games on Facebook to get those extras, thanks to Facebook Connect. You’re not on Facebook? Don’t worry, you can still chat, play and have fun on Good Camel Games, being on Facebook is not mandatory at all.

New Game: Longest Word – A new game called “Longest Word” has been added. The goal of the game is to form the longest word you can think of with the given letters. If you like Scrabble, you will love that game. You also get to choose the dictionary to be used for validating words. At the moment, 4 dictionaries are available: English, French, Spanish and German. Feel free to suggest more languages if your native language is not in that list.

New Error Handler – In our effort to provide quality services, we have a mechanism that notifies us when an error occurs on Good Camel Games. That mechanism will also alert us when an error occurs on the client side (in the browser). In order to do so, the browser sends a request to the server with the details of the error so we can investigate an fix the issue. That part of the server that receives errors is called the Error Handler. In order to make it robust and always available, the Error Handler has been re-implemented as a Google App Engine project and is now hosted on Google servers: Good Camel Games – Error Handler. This way, the Error Handler will be able to handle errors even if Good Camel Games goes down or is experiencing other major issues.

Play Alone – We’re also proud to announce that most games can now be played alone. So you don’t have to wait for an opponent to have fun. To play a game alone, just start a new game as usual and click the “Play Alone” link. Games are more fun when played with somebody else, but sometimes nobody is available. You can also use that feature if you want to practice your skills at a game.

Send Link – In order to facilitate playing with people you know. We added the “Send Link” feature. So you can start a game as usual and use the “Send Link” feature to copy and send the link (url) to the person you want to play the game with. That person does not need to be on Facebook and does not even have to know about Good Camel Games. When that person clicks the link, you will see him/her as an opponent wanting to join the game and you just have to “Accept”.

That’s all for now! Feel free to suggest more games and features! Good Camel Games – Fun Multiplayer Games

Good Camel Games is a GWT web app and it is therefore a very JavaScript heavy application. The problem is that non-JavaScript enabled users just cannot use my application, so for them, I was showing a page saying they cannot use the application and some information about the application and what it actually does (that info was also available for JavaScript enabled users, but somewhere else). Big mistake! Showing different content to JavaScript VS non-JavaScript users violates Google’s quality guidelines because it is often used as deceptive technique to fool search engines. That technique is known as cloaking and Google has mechanisms to detect that.

Fortunately, you can request reconsideration of your site. So I made some changes that hopefully fixes the problem, created a short video explaining the details (not required, but I think it is easier to understand) and submitted the reconsideration request to Google.

Here is the video: My GWT Application is not Indexed by Google

Here is the message I included in my reconsideration request:

Hi,

My web application has been indexed by Google for a short period of time, but it is not indexed anymore. I think the problem with it is that it is not compliant with Google’s quality guidelines.

I made a short video that explains why I think it is not compliant with Google’s quality guidelines and also explains what I did to fix the problem. I made a video because I think it is easier to explain with images and also to help other people not to do the same naive mistake. Here is the video:

http://www.youtube.com/watch?v=WIHTUJBawnI

Basically, the explanation is the following: My web application is built on GWT and requires JavaScript enabled in the browser. I think the problem with it is that does not present the same content to JavaScript VS non-JavaScript enabled users. I fixed it by adding an extra step (at the beginning) for JavaScript enabled users which shows pretty much the same thing as non-JavaScript enabled users. But see the video, it will be easier to understand and I think it demonstrates my good faith.

To answer your question about SEO, no I did not use any SEO firm, I did this mistake all by myself.

Regards,
Jonathan Demers

Here is the automated response I received:

We’ve received a request from a site owner to reconsider how we index the following site: http://games.goodcamel.com/

We’ll review the site. If we find that it’s no longer in violation of our Webmaster Guidelines, we’ll reconsider our indexing of the site. Please allow several weeks for the reconsideration request. We do review all requests, but unfortunately we can’t reply individually to each request.

As suggested, I also asked for a site review in Google Forum: My Website (Web Application) is not in Google index anymore. Can somebody review my site?

The only thing is that it can take several weeks for a reconsideration request and you are not notified when the reconsideration process request is over; or your site is OK and added to the index, or it is still not OK and not added to the index. So hopefully, you won’t do the same naive mistake i did.

Anyways, I’ll update the post when I get more info.

… to be continued

UPDATE – November 18 2009: I can see Good Camel Games in Google index tonight. Wow, that was fast. Thank you Google!

UPDATE – November 20 2009: Yesterday (Nov 19), I actually received a notification from Google telling me that my site has been reviewed. That is nice, I did not expect that. And my application is still in Google index. Here is the Notification:

We received a request from a site owner to reconsider how we index the following site: http://games.goodcamel.com/.

We’ve now reviewed your site. When we review a site, we check to see if it’s in violation of our Webmaster Guidelines. If we don’t find any problems, we’ll reconsider our indexing of your site. If your site still doesn’t appear in our search results, check our Help Center for steps you can take.

This is an automated message from Webmaster Tools. Please do not reply to this email. For help, please visit http://www.google.com/support/webmasters.

<script type=”text/javascript”>
window.onerror=function(message, url, line) {
window.location.href=’/error?msg=’ + escape(”Error: ” + message + ‘\nUrl: ‘ + url + ‘\nLine: ‘ + line);
return true;
};
</script>

And then, on the server side, I just have a servlet that handles “/error” and sends me a notification with the content of request parameter “msg” plus some other info about the client.

So I received that error notification (just pasting the relevant parts here):

From: googlebot(at)googlebot.com
user-agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
msg: []

After looking at it carefully, I realized that GoogleBot does not actually execute JavaScript. You can see this because the “msg” parameter is there but empty, which means GoogleBot sent a request to “/error?msg=”, without the parameter value (”Error: …”). So GoogleBot just looks in the JavaScript and  takes whatever resembles to a URL and tries to index that URL.

After some researches on the web, I found I was not the only one:

• This guy from seomoz.org having unexpected parts of his site indexed: New Reality: Google Follows Links in JavaScript.
• This guy explaining  “Stealth Links” and mention JavaScript URLs as one of them: Stealth links and Googlebot.
• This guy having interesting stories about JavaScript crawling by GoogleBot: GoogleBot Can Crawl JavaScript With Clean URLs.
• This guy who got  call from GoogleBot: Googlebot called me on my Timpani Live Help.
• All in line with a Larry Page quote: “We added the ability to search for code in more than 40 different programming languages [...]“ from Google Works on Unified Search Engine.
• This guy talking about the implications of that new feature in GoogleBot: Google Crawls Javascript, News at 11!

To be very sure, I added another test on the home of Good Camel Games:

<script type=”text/javascript”>
function thisFunctionIsNeverCalled() {
window.location.href=’/this_page_does_not_exists’;
}
</script>

If GoogleBot hits the URL “/this_page_does_not_exists”, I will receive a 404 error notification. I added that code today, I will update that post when I have the notification. Hopefully, Google will have some day a true JavaScript engine to crawl JavaScript heavy applications.

UPDATE – November 20 2009: I confirm GoogleBot tried to access the page “/this_page_does_not_exists”. This confirms GoogleBot does not executes the javascript, it just checks for URLs in it. Here is the 404 error notification I received (just pasting the relevant parts here)

From: googlebot(at)googlebot.com
user-agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Request URL: http://games.goodcamel.com/this_page_does_not_exists

And actually, I you can also test your Trackback/Pingback using this blog post. I will accept any Trackback/Pingback, as long as it is not spam. However, it may take up to 24 hours for me to approve it, so just be patient

And if you need to be convinced that Pingback/Trackback are important to your blog, here is a great article that explains why: Link Marketing: Pingbacks vs Blog Comments

… well, everything seems pretty good at this point.

What the client sees is very similar, but I have to admit that the admin looks much nicer.

• A corporation wanting its employees to give anonymous and interactive feedback about things that could be improved in the organization.
• A family member expressing his/her views on a taboo subject.
• An individual in a group wanting to change things and trying to create a momentum.

USpeak firmly believes Freedom of Speech is for everybody and encourages people to freely speak their thoughts. For more information, see USpeak FAQ.