They are not alone in that. The "Archiveteam", a different organization, not to be confused with archive.org, also doesn't respect robots.txt according to their wiki: https://wiki.archiveteam.org/index.php?title=Robots.txt
I think it is safe to say that there is little consideration for site owners from the largest archiving organizations today. Whether there should be is a different debate.
It seems like the general problem is that the original common usage of robots.txt was to identify the parts of a site that would lead a recursive crawler into an infinite forest of dynamically generated links, which nobody wants, but it's increasingly being used to disallow the fixed content of the site which is the thing they're trying to archive and which shouldn't be a problem for the site when the bot is caching the result so it only ever downloads it once. And more sites doing the latter makes it hard for anyone to distinguish it from the former, which is bad for everyone.
> The "Archiveteam", a different organization, not to be confused with archive.org, also doesn't respect robots.txt according to their wiki
"Archiveteam" exists in a different context. Their usual purpose is to get a copy of something quickly because it's expected to go offline soon. This both a) makes it irrelevant for ordinary sites in ordinary times and b) gives the ones about to shut down an obvious thing to do, i.e. just give them a better/more efficient way to make a full archive of the site you're about to shut down.
What an absolutely insufferable explanation from ArchiveTeam. What else do you expect from an organization aggressively crawling websites and bringing them down to their knees because they couldn't care less?
ArchiveTeam (which is not the Internet Archive) aggressively crawls websites because they care a lot, because the website in question is about to go away.
Heck, I'd say as caring goes, ArchiveTeam cares more than the owners of the website, because in the ideal shutdown, the owners provide the data instead of forcing people to scrape it if they want to retain it after the site shuts down.
They also crawl aggressively when the site is not in danger. They crawled my MediaWiki because someone else input the site in their bot and it overloaded the PHP process. I know that archiving is important but please, not like this.
I'm curious to hear about examples of where this has happened. Because ArchiveTeam also has an important role in rescuing cultural artefacts that have been taken into private hands and then negligently destroyed.
ArchiveTeam definitely do not intend to kill websites with too fast crawling, but definitely have done that unintentionally and always will stop/slow the crawling when it happens.
Even the distributed crawling system has monitoring and controls to ensure it doesn't kill sites.
As and engineer, sometimes getting started is the hardest part. But there are high quality, open source templates for almost anything in the documentation process available. I wish initiatives like The Good Docs Project would be more popular with engineers.
I really tried to be open minded and clicked your link. You lost me at:
> By combining Web2 (community interfaces) and Web3 (smart economies), we can launch the next generation of applications — which we refer to as Web5.
This is not how versioning works. It looks more like marketing coming up with the highest number possible without a meaningful increase in value delivered. (Remember the old
PC magazines with 999+ games on a CD Rom?)
Have driven in Waymo and Cruise quite a hit. Don’t wanna beat a dead horse, so let’s talk about Waymo only: It improved every time from when I first got access to today, where a lot of people can use it now. I even had my visiting parents riding in Waymo cars a couple of times and they felt saved and loved it, and they are not really early adopters of anything :)
When in SF I always prefer Waymo over any other form of ride service or ride share.
Disclaimer: Google LLC employee. No relations to Waymo, just a fan of self driving in general.
I genuinely thought this link was about escaping a standardized steel shipping container, something I recently had to seriously consider. In that regard a disappointing click.
I also wrote my own docker-like containerization code for educational purposes a while ago, so container has both these meanings for me. Yet, me brain was expecting a physical escape story. Brains are funny!
Here in the U.S. west coast mountains some land owners started controlled fires on their property to get rid of the stacking fuel naturally while preserving the bound minerals and helping the large redwoods and sequoias to fend of contenders. I have no idea how they managed to get a permit in this area where officials and population ate crazily scared of these natural processes given that uncontrolled fires make the news every year.
Also, a second generation redwood forest looks very different from an undisturbed one I recently learned from a forest guy who walked with me. He was reading the forest like a book. Very impressive. Turns out, my forest is a second generation and I should maybe take down a few redwoods, something I considered morally wrong before the walkthrough.
Prescribed burns (intentional, hopefully controlled, fires) are increasingly common in modern forestry and wildfire prevention. They're done in Canada and Australia, and from a quick search, it was brought back into practice in the 1990s in the USA.
In South Australia, they're very much monitored and controlled. In the Northern Territory however, it is not uncommon in season to see fires burning beside the highway in a national park, and no one anywhere nearby monitoring them. Well, that is besides birds who hang around to nab fleeing insects and other creatures. Birds have been known to carry burning twigs elsewhere to spread the fire too.
Coming from the former region and visiting the latter, it's quite jarring. But that (now called) mosaic burning has thousands of years of history. You can easily see how it advantaged the early occupants of the area: reduce overall fuel load, flush out animals to hunt, and clear annoying tall, dry grasses which would be miserable to live amongst and walk through.
In Western Australia they are a racket. Land is burned for burning's sake as an area of required burning is defined and people rush to hit a quota. Studies are showing that land left untouched for years is actually less flammable than land in the medium turn after burn-off. Additionally, the effects on air quality cannot be discounted either. It is a politicized issue and people are terrified of being seen as not doing anything to combat bushfires that would otherwise threaten homes.
this comment is useful to show how opposed various parties can be.. I have > forty recent, peer reviewed forestry sciencepapers on this topic here in California, where I share them with others who want to educate themselves on the topic.. here we share some characteristics with some parts of Australia regarding wildfires.. in recent years, the severity and scale of some fires have shattered previous records.
My understanding is that California air pollution regulations make it incredibly difficult to burn. The fact that, if you don't do a controlled burn it will result in an much larger uncontrolled burn later is not taken into account by the regulations.
It's a bit more complicated than that: the California Air Resources Board sets statewide rules but the permitting for prescribed burns and the final responsibility for air quality lies with the 35 air districts. They already have the power to ignore many air quality regulations when granting permits but their politics are all over the place and there's tons of locally driven NIMBYism that's more influential than it would be at the state level. In some districts prescribed burns are a lot easier than in others.
CARB has been researching the issue of prescribed burns for decades and ever since the mid to late 2010s is completely on board, but the air districts are slower to follow. On top of that, the difference in land management priorities between the National Forest Service, BLM, and the State of California complicates everything. One of the best arguments for the Federal government divesting of its land in the west is to allow the states to better manage their own resources (states' rights comes full circle).
after the 2018 season, Gov Newsom did oversee a series of comprehensive settlements between major parties regarding cost, authority and procedures. Secondly the long-standing CalFIRE lead was terminated.
Yup - sooner or later mother nature wins. And those larger burns? Instead of being beneficial they actually kill the trees. It's what happened to Yosemite in the 2000's and killed thousands of acres of trees.
People like to blame various west coast states for this, but do note it's mostly federal land, managed by federal employees. East coasters tend to not comprehend how much federal land there is in the southwest/west; watching tourists is fun.
Yea I'm not blaming the state governments because I don't know who's responsible for burns, just pointing out that the end result is the west coast is a tinderbox relative to the rest of the country
> You don't need to do a controlled burn if the wildfire already did one for you.
That seems to get cause and effect backwards. Had California done more controlled burns, then they wouldn't have had nearly as many disastrous wildfires.
California would still have lots of disastrous wildfires regardless of controlled burns, because the chaparral ecosystem is spark-limited, not fuel-limited. Unless you limit your comments to forest ecosystems, which is only half the story, and very little of the urban-wildland interface.
Controlled burns is a very old practice. So old, in fact, that Native American tribes have used it for centuries to prevent catastrophic wild fires in North America.
The (US) National Park Service disagrees with you, so much so that in some places they hire native elders to help them with controlled burns. They call the practices "cultural burning". They were done for many purposes, over millenia.
> The (US) National Park Service disagrees with you, so much so that in some places they hire native elders to help them with controlled burns. They call the practices "cultural burning".
But that can only be justified by a raw appeal to public relations. No native elder has any relevant experience today.
The small tribes of the northwest might have used fire for clearing land in their immediate area but there is no evidence that they were managing hundreds of thousands of acres of forests in order to reduce large wildfires.
According to the research done by Charles Mann in 1491, the indigenous population used fire to drive prey into hunting zones. I believe he even presents the theory that is what created the grassland plains across the central US. He presents quite a bit of evidence from primary sources.
I’m not a historical scholar and so maybe his evidence is “bunk”, but his work seems to be very well received in academic circles (unlike say Graham Hitchcock who is seen as more of a Malcolm Gladwell type).
Most of the area in between the Mississippi and the west coast doesn’t get enough precipitation to support trees. I don’t think that was generally understood until Powell’s survey a couple hundred years later.
Yeah it’s one of those things that kind of sits uncomfortably in the “native Americans were wise nature wizards and we have to unlearn our toxic western industrial capitalist beliefs in order to rediscover their hidden mystical wisdom to save the planet” territory
Like it’s a good story, and it’s true to some degree, but the pageantry around the language people use when treating it is… I dunno it just still sounds like gross Cowboys-and-Indians prose.
There is very little original "old growth" forest anywhere in the Continental US. It is usually in small parts of hard to log areas like around small streams, soft ground blocking how to remove the logs and certain slopes and hills.
Sometimes those landowners doing prescribed burns don't always do them in the best conditions. The Estrada Fire was near the Santa Cruz Mountains east of Watsonville. I've been to the property several times over the years and it has some beuatiful redwood groves including an albino redwood or two.
At least part of that is the climate. A forest in a tropical zone is so efficient at processing nutrients that the soil beneath a forest is almost nutrient-free; the nutrients are always moving from plant to plant. If you take off the forest, what is left is not very hospitable and erodes easily.
Temperate forests accumulate humus; remove the trees and there are nutrients sitting there waiting to foster new growth.
I'd question that definition of "pristine". The old-growth temperate rainforests of the Olympic Peninsula in Washington, have only existed for about 10-12,000 years; that's about how long it's been since the entire area was covered by an ice sheet.
Here's an overview that has a timeline and some jumping off points if you're interested in learning more. What happened as the ice retreated varied by area and some of the valleys in particular have been studied to see what species arrived, in what order, and to investigate delays by some.
> In comparing the various palynological sources for Northwestern Washington and surrounding regions, it is clear that the vegetation history varies at least in its details from area to area. For example, Heusser (1978:1576) notes that treeless conditions persisted longer after glacial retreat in the Hoh Valley than in the northwestern corner of the Olympic Peninsula.
> ...the arrival of coniferous trees in this area was apparently delayed by aridity until sometime between 11,000 and 9,600 yr B.P.
This makes me want to see designated areas for old-growth forests to re-establish, and yet there’s almost no chance that the people living in those areas 2000 years from now will have continuously held the same values and kept the project going.
Plenty of people in California are doing this. Basically they buy a large amount of acreage up where nobody cares (mostly Mendocino county or Humboldt county, but sporadically throughout Sonoma County / Marin County/ rest of the Bay) then spend about a decade putting the land into a trust, and marking each separate trees to make sure people don’t poach them. The largest one is the ‘Save the Redwoods’ league, but I’ve spent many days hiking through redwood preserves just from some random person who died and made a land trust as their legacy.
People 2000 years from now will also be enamored with Redwoods (provided they still exist), they’ve been highly regarded for thousands of years already, and they will for thousands more.
The idea of using nuclear waste to protect pristine natural environments is a good one. It's the most effective one we have today. After a few generations I heard animals aren't affected anymore, it also depends on the kind of fallout.
I can second that. I published a Copperfield illusion analysis and got approached to take it down from other magicians (not Copperfield himself). I left it out there in the open. For me, trying to crack the puzzle is part of the fun and why I spend money seeing magic shows multiple times.
If only... Despite providing a useful service, they are not as nice towards site owners as one would hope.
Internet Archive says:
> We see the future of web archiving relying less on robots.txt file declarations geared toward search engines
https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
They are not alone in that. The "Archiveteam", a different organization, not to be confused with archive.org, also doesn't respect robots.txt according to their wiki: https://wiki.archiveteam.org/index.php?title=Robots.txt
I think it is safe to say that there is little consideration for site owners from the largest archiving organizations today. Whether there should be is a different debate.