zenkovu пишет:Как получить список всех страниц сайта программой под linux ?
уже была такая тема.
Выводы в двух словах:
1. для динамических сайтов это невозможно в принципе.
2. для статических сайтов
man wget
Recursive Retrieval Options
-r
--recursive
Turn on recursive retrieving. The default maximum depth is 5.
ну и дальше читайте.
На практике очень полезно применить это:
--wait=seconds
Wait the specified number of seconds between the retrievals. Use of this option is
recommended, as it lightens the server load by making the requests less frequent. Instead
of in seconds, the time can be specified in minutes using the "m" suffix, in hours using
"h" suffix, or in days using "d" suffix.
Specifying a large value for this option is useful if the network or the destination host
is down, so that Wget can wait long enough to reasonably expect the network error to be
fixed before the retry. The waiting interval specified by this function is influenced by
"--random-wait", which see.
--waitretry=seconds
If you don't want Wget to wait between every retrieval, but only between retries of failed
downloads, you can use this option. Wget will use linear backoff, waiting 1 second after
the first failure on a given file, then waiting 2 seconds after the second failure on that
file, up to the maximum number of seconds you specify.
By default, Wget will assume a value of 10 seconds.
--random-wait
Some web sites may perform log analysis to identify retrieval programs such as Wget by
looking for statistically significant similarities in the time between requests. This
option causes the time between requests to vary between 0.5 and 1.5 * wait seconds, where
wait was specified using the --wait option, in order to mask Wget's presence from such
analysis.
а то забанят.
Ну и уровень рекурсии тоже, а то оно вам весь интернет скачать может. (не сможет, но попытается).
Карусель разнесло по цепочке за час
Всех известий — конец
Да, весна началась!
(всё к лицу подлецу, как родному отцу, не рассказывай, батя, и так всё пройдёт)
