Today, all technological processes do not cease to speed up the already rapidly growing pace of development of the digital age. This is especially clear in the IT industry, where applications of new tech have reached all aspects of our daily life. The time frame for the implementation of new solutions is shrinking at such a speed that sometimes you have to adapt to the new realities of the market very fast. This rush can often hurt your business or professional skills.
At the very beginning of its history (Q4 2017), the Revain review platform focused on the fundamental decision when choosing a convenient environment for developing a web application or Single-Page Application (SPA).
This choice was determined by competitive advantage, since any user interaction with the platform should be as convenient and dynamic as possible: the user receives an instant response to any changes in the area of his screen, without reloading the page.
Popular frameworks such as Angular and Vue.js, as well as the React library, allow the developers to easily create products of the highest quality. You may remember some of them: Facebook, Twitter, Gmail, Google Maps, AirBNB or Netflix.
However, aside from visual effects and ease of use of the product, there is another method for developing a content project - organic traffic from search engines!
Content is King
This rule is the foundation of any effective SEO strategy. If the reviews are useful, people will read them. And in order to consistently increase your outreach, you need to work closely with search engines.
The task becomes more complicated if:
- Most of the content is user-generated
- The platform supports multiple languages
- The number of pages increases by tens of thousands every month
- Information on landing pages is updated frequently
- You don't know how Search Engine SPA works in practice
Of course, Google has been saying for quite some time that JavaScript is not a problem for their engine. For example, in 2018 they provided dynamic rendering as an indexing improvement option for SPA. We at Revain welcomed this update!
And indeed, the indexing problem was solved. But this turned out to be only the beginning of a complete change and testing of almost all the architectural principles of the platform.
Cons of dynamic rendering for multi-page dynamic sites
Our experience shows that using this method is great for small projects where the number of pages and the information on these pages does not change too often.
You simply form a list of all the main crawlers and provide them with the necessary content for crawling and indexing.
For example, here is a list of crawlers we started with:
revainbot | googlebot | yandexbot | gigabot | bingbot | naver | duckduckgo | facebookexternalhit | facebot | linkedinbot | slurp | baiduspider | twitterbot | screaming | semrushbot | semrush | seomoz | yahoo | ask | mediapartners-google | adsbot-google | ads-google-mobile | bingpreview | msnbot | adldxbot | duckduckbot | applebot | linkedin | rogerbot | dotbot | mj12bot | ahrefsbot | ia_archiver | ahrefssiteaudit
In the end, everything seems to work: users enjoy the technology, and search bots index the necessary pages.
Only one “small” problem is yet to be solved - search ranking.
How Dynamic Rendering works. Source >>>
1. Using a third-party service as a pre-render
The problem with third-party services is that we can’t change their functionality. For example, in our case, using prerender.io led to caching a huge number of unnecessary pages for indexing.
At that time, the Revain review platform already had over 10k pages and about 100k more pages that we did not want to be indexed by search engines.
Even after contacting Prerender.io support service, we were unable to implement priority caching, focusing, for example, solely on our sitemap or the possibility of using black and white lists for certain types of URLs.
Taking into account the perspective of the development of the platform, it also became clear that with a significant increase in the number of new pages, their indexing time will only increase.
5xx error problem
We also ran into the issue of 504 errors that have been occurring intermittently on our platform. It turned out that prerender.io tries to re-cache the broken URL 3 times within 30 minutes, and if this URL returns a 504 error all this time, it will be removed from the cache.
Thus, some of our pages were constantly falling out of the search engine index until the pre-render came back to them again, i.e. time for re-indexing constantly grew.
We also failed to optimize this rule.
Moreover, the development of the platform did not stop, and we were certainly not immune from further problems with 5xx errors in the future. All this had a very negative effect on SEO.
Still, it is worth noting the excellent work of prerender.io technical support. The guys really tried very hard to delve into each of our problems and deal with it.
But, unfortunately, in our case, we had to develop a more flexible solution for moving forward, and we had to do it fast.
2. Our own pre-render
Having our own service certainly helped us deal with a lot of previous issues, but this solution still felt like a temporary fix.
Difficulty in optimizing caching processes
Even with a priority queue, the native pre-render couldn't handle the many pages where content is updated frequently. The recaching delay caused a delay in the reindexing time (at some point we even started using forced batch indexing, which, however, did not help much).
Because of this, there were also unpredictable risks associated with cloaking, which are perceived differently (or may be perceived in the future) by secondary or regional search engines, such as Yandex, Baidu, Bing, DuckDuckGo, etc.
As an international product, Revain simply couldn't operate right while ignoring other sources of traffic.
Bilateral analytics and development
Diagnosing the operation of the platform is quite complicated due to the need to analyze two parallel processes simultaneously: how a search crawler sees a page and how a regular user sees it.
Also, developers had to support 2 different applications at the same time - desktop and mobile versions.
Cross-browser and cross-platform
Not all versions of browsers and not all phone models allow the users to render the needed JavaScript. Which, among other things, drives away potential authors or readers from developing countries.
Off-page SEO
Using dynamic rendering also does not allow interacting with most analytical services that cannot render JavaScript. For such services, the technical condition of your SPA will always look critical, as their tools will only see one page and will most most likely perceive it as completely blank.
And sites like alexa.com would not allow you to pass verification.
Page loading speed
We have also been unable to optimize Google PageSpeed performance well enough all this time until we finally came up with the necessary solution to the full range of these fundamental problems.
Search for solutions
Since December 2018, we at Revain have been trying to understand the basic principle of how SPA interacts with search engines. Why doesn't Google "love" us so much? And what else can we do to make our relationship much better?
Social networking groups like the JavaScript Sites in Search Working Group have often been advised to definitely look into SSR or Server-Side Rendering.
And some of our colleagues, who were working on similar large-scale SPA projects, were simultaneously developing additional HTML sites for search crawlers.
Further, in February 2019, an overview article on all types of rendering is published: Rendering on the Web, by Jason Miller and Addy Osmani.
Also in February of the same year, practical video tips on JavaScript SEO appeared on Google Search Central.
In simple words, finding the right solution no longer seemed so obvious. There was a need for a quality third-party auditor, whose practical experience would help us find the only option that works for us.
Content is King again!
It is not difficult to find a huge amount of unique and useful information on the Internet. But this was not entirely our case.
For almost 2 years, we have studied the experience of various experts in the field of Technical and JavaScript SEO and communicated with them for possible interaction. We tried to optimize the work of dynamic rendering. We analyzed existing SPA projects and the technologies they use. Yet in the end, we received the most useful information, as well as high-quality support and expertise from these wonderful guys: https://www.onely.com/
It’s not easy to imagine the fate of the development of the platform if, according to the main search queries regarding our SPA problems, we would not have found detailed guides with practical recommendations from the Onely team.
SSR with (Re)hydration
This is the solution that we have been looking for more than 3 years, and which cost us more than a year of implementation, managing to simultaneously develop many product tasks for the platform.
Server-client spectrum. Source >>>
As a result, we got a significant boost for Revain: organic traffic and good ranking of the landing pages in the SERPs and this is only the beginning!
Organic TOP 10 Keywords Growth. Source >>>
Conclusion
Naturally, promoting and developing a project like Revain takes more than just a transition from dynamic rendering to SSR with rehydration. There is still a lot of work ahead of us, but at the moment, it is safe to say that we have overcome one of the most significant and fundamental challenges in the history of the first blockchain review platform.