[routing-wg] RPKI Validator 3 Disk Space Requirement

>> the rpki-validator.h2.mv.db file is already >22GB in size, I assume
>> there is no way to reduce the disk usage requirements via a
>> configuration change?
>> 
>> How big will the db file become?
> 
> There is a config setting regulating how often the validator will
> remove the "garbage", i.e. unused objects that it still stores but
> they are not really used but kept just in case (no connectivity to
> download updated records, broken manifests, CRLs, etc.). The setting
> is called "rpki.validator.rpki.object.cleanup.grace.duration" and by
> default it is set to "P7D", i.e. a period of 7 days, which is pretty
> conservative. You can set it to much smaller periods, 1-2 day or 12
> hours and still have reasonable behaviour. There is another parameter
> “rpki.validator.validation.run.cleanup.grace.duration" with the
> similar meaning, but it's less crucial when it comes to storage
> space. Also Validator version 3 is lighter than 2 in a sense that it
> consumes less memory, but it can only do it by storing more data on
> the disk.
> 
> I hope this helps.

thanks for your reply.

I changed it to:
rpki.validator.rpki.object.cleanup.grace.duration=P1D
in conf/application.properties, restarted the validator and expected some reduction in 
disk usage after a day or so, but the db folder uses almost 50GB now..

What disk usage should I expect / prepare for?

There are also some errors shown on stdout (details see bellow).

Since there doesn't appear much activity on the RPKI validator 3 issue tracker
a general question:
What is the general maintenance status of RPKI Validator 3?

thanks,
nusenu



ERROR   pool-1-thread-1                         o.h.i.ExceptionMapperStandardImpl               HHH000346: Error during managed flush [org.hibernate.exception.ConstraintViolationException: could not execute statement]
ERROR   pool-1-thread-1                         o.s.s.s.TaskUtils$LoggingErrorHandler           Unexpected error occurred in scheduled task.
org.springframework.dao.DataIntegrityViolationException: could not execute statement; SQL [n/a]; constraint ["RPKI_OBJECT__SHA256_IDX ON PUBLIC.RPKI_OBJECT(SHA256) VALUES (X'e17d74b2082b15701d300e24d97a3d7b0eb081d59034c6d026b9555b0509bc0e', 3443950)"; SQL statement:
/* insert net.ripe.rpki.validator3.domain.RpkiObject */ insert into rpki_object (created_at, updated_at, version, authority_key_identifier, last_marked_reachable_at, serial_number, sha256, signing_time, type, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [23505-197]]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute statement
        at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:259)
        at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:225)
        at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:540)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:746)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:714)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:532)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:304)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
        at net.ripe.rpki.validator3.domain.validation.RpkiRepositoryValidationService$$EnhancerBySpringCGLIB$$3380f99b.validateRsyncRepositories(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.exception.ConstraintViolationException: could not execute statement
        at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:112)
        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
        at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:178)
        at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3032)
        at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3547)
        at org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:600)
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:474)
        at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
        at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
        at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1437)
        at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:494)
        at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3245)
        at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2451)
        at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:473)
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:156)
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$100(JdbcResourceLocalTransactionCoordinatorImpl.java:38)
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:231)
        at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:68)
        at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:536)
        ... 21 common frames omitted


-- 
https://twitter.com/nusenu_
https://mastodon.social/@nusenu

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: </ripe/mail/archives/routing-wg/attachments/20181009/cc5a3918/attachment.sig>