You would like to know what form of password encryption is used by DNN.
DNN user registration is built upon Microsoft's MembershipProvider classes that ships with ASP.NET, which also handle the storage and retrieval of passwords. Please refer to Password Formats for more information.
As the AspNetMembershipProvider is not a DNN-specific component but a Microsoft component, we're unable to provide support regarding the details of how it encrypts users' passwords.
You can find the information in ASP.NET MembershipProvider -- How exactly does it do encryption? useful as it provides specific details regarding the implementation, or refer to Microsoft's How to: Use the ASP.NET Membership Provider documentation along with this StackOverflow post. This provider uses SHA1 as the default hashing algorithm.
It would also prove useful to review the MembershipPasswordController.cs file and related files for the actual application code involved.